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SYNOPSIS 

* Jhe objective of this stu.dy was the development and validcition of 
an eff€Ctive^ method for obtaining post /ormal training feedback information 
suitable for ^ use throughout the Navy training syst^em. The continudiis ^ 
collection this information from, operational units concer^iing the job 
performance of school .graduates is vital for maintaining up-to-date, > ^ 
effective training programs. * Although a number of unrelated approaches 
for obtaining. post formal training feedback exist in d'iff^reat Navy 
schools, tjiere is no standard, systematic method for obtaining such 
data. To fill this gap in the evaluation of trailing programs-, the 
Chie^ of Naval Education and Training (CNET), N-34, assigned to the 
Training Analysis Bnd ^Evaluation Group (TAEG) the task af developing 
s^ch a method. ^ ^ , , ' • ^ , 

1 A prel i mi nary examination of ekistin^ feedback techniques indicated 
that the mail -out questionnaire and the perspnal interview were the, 
general approaches most suitable for this purpose. ;The /questionnaire 
method has the potential for inexpensively providing reliable data from 
the widest, possible range of -operational units, ilowever, low return - 
reites and data of poor quality have been the more typical results when ^ 
this feedback method has been used by Navy schools. Prior to selection 
?ind development of a feedback method for Navy-wide use, i*t was essential 
to resolve many questions about the effectiveness of mail -out feedback 
methods and to compare , their performance wjth thdX of more Ijighly esteemed 
personal interviews, to do this, an empirical comparison of mail -out 
and personal interview feedback methods was undertaken in a mi,litary 
training setting where m&ny trainitig problems had already been identified. 
Different feedback methods could thus be easily compared on the accuracy 
with which, they disclosed these- known, training problems.. 

The Radfoman "A" School was selected for this development and 
comparison of feedback methods. The school had recently undergone a 
major curriculum revision based on a job, task analysis. The new curricu- 
lum had not yet been o'mplemented- and the many training problems identified 
in the old curriculum would* serve to measure the effectiveness the ^ 
different feedback methods. In addition, the large' number of graduates 
produced by the school provided the study team with the opportunity .to 
test a number of different mail-out Instruments and admin'tstration^ 
techniques. • * , , * 

Questionnaires were mailed to ^996 recent Radioman School graduates 
and 59Crssupervisors of such graduates. 'The names and locations^ of thes| 
personnel had been confirmed by letters returned from the operational 
'units, prior to questionnaire mailiilg. Three .different questionnaire 
formats were evaluated. The first was a longfornj (134 items)^made up 
of spetQific training and job task statements. ^^e second was "a: short ^ 
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fzmji (4-5 Items), made from a list of general job task statements that ' ^ 
subsumed £^e specific job tasks* of the long form. 6n both forms, the ' \ 
respondent waS' asked to rate (1) the frequency with which the trainee *' - 
performed the task, , (2) the critical ity .of the task, and (3)*howwell 
the trainee ^could perform the task upon arrival at the^ operational unit. 
The third instrument was a card-sorting technique in wh'ich the 1.34 items 
ffom, the long questionnair^e were^^ printed' on small cards to- be sorted by , 
th6 respondent into categories related, to task frequency and to the . ' • 
trainee's ability, to perform the task. ' In addition to mail-out procedures^, 
5. group of 59 trainees and 37 "Supervisors ^ were interviewed face-tb-face, 
using a structured' interview based on the long questionnaire. 

The data obtained by eac^ method were analyzed for th?^ accuracy arid 
xojTiprehensiveness with which the rating scale data identi^ed the knpwn 
training problems that had existed for the graduates. Rettirn rate 
statistics were also computed for the different mail-out instruments as . 
^well ^s statistics on time'until instrument return. Separate analyses 
on these variables were conducted for^l) trainees and supervisors, (2) 
persons who expected the questionnaire an^ persons who did not, (3). 
graduate's With different' times from graduation prior to evaluation, (4) 
•graduates with different class standings, and (5) graduates with different 
xiuty stations. The last variable was investigated because many of the • 
problems of the old Radioman *'A" -School curriculum were related to the e 
inappropriate training. on shipboard systems -givgn to' persons assigned to 
shore stations. % , . ^ . • , . 

" .-^ ^.^ 

All of the instruments and procedures included in the sttidy were^at ; 
least moderately successful in identifying th'e known trainirig problems 
as well as identifying successful training. The Iqng qiiestidnjiaire 
based on' specific job task statements 'was particularly effective and 
provided data that were nearly identical tq the data obtaiped from the 
personal interview method. These data were highly reliable (i.e.,, 
consistent from one respondent to another^, and-they accurately and. 
comprehensively identified the shortcomings known to exist in the old 
Radioman "A" School curriculum.. .Supervisory personnel provided somewhat, 
better quality information than trainees, although both were excellent' 
sources of ddta about 'training problems •and both should be called on to 
provide^ feedba^ck information. V - ' 

The overall rate of feturn for instruments was satisfactory to meet 
training needs. The rate of return was 59 percent for supervisors and 
31 pelrcent for trjainees. The return rate for short form questionnaires, 
was significantly greater than for the long form with>ost of the differ- 
ence between the instruments contribqted by the trainees*. Despite ? 
higher rate of return, the short fdif^n failjsd to identify many specific 
training problems that were disclosed by the lonjg questionnaire.'' 
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An expectation or "set" to receive the mail -out instrument was 
found to significantly iliiprove' the return rate of trainees. Such a 
"?et" should'be established in th^ graduate,, either as it was done in 
|fiis study 6y preceding the quest! onna-^re with a letter or by contacting 
hdm prior to* his,. departure fr^m school. The optimal tifne lapse between 
graduation^and administration of the feedback instrument was ^determined 
to be. five td seven months. . . , * - - • 

' Because of the excel! en^rea^te^^obtained in this study with the 
questionnaire based on^speoBMMHPsks, it is recommended that the 
CNET adopt this particular llJIpment format and associated procedures 
as the major Tiiethdd for'obtainTiig post fonnal training feedback information 
in Navy schools. . The recommended feedback/curriculum revision cycle 
takes 18 months plus the duration of the course. A minimum number of 
personnel are required to develop and acftninister the instruments and to - 
analyze the data. Procedures ar^ straightforward and require no particular 
training of the person or persons assigned to carry them out. These 
procedures are also described in a' companion report (TAEG "Feedback 
Manual," June 1975) written specifically for "the training , personnel who 
will be primarily responsible fdr implementing them. 
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/SECTION I 



.. INTRODUCTION . - 

• " > • • . / , ■ ° 

A trailing program is effective only to the Extent that it meets 
the fieeds of the Fl.eet aijid other operational units for qualified personnel! 
These^ needs 'must be given the highest priority in the design'of training 
programs and also in the assessment of 'these programs. Course design ' 
based on job task analysis meets this critericjft, e^nd procedures already 
exist for this purpose (Butler, 1972; £ranson, '1974"; Rundquist, 1970; 
Smithj 1971). However'there is a gap in the evaluation of training 
programs, since standard and systematic procedures do not ex-^st for 
provfding -sdhools with ipb performance data°on their, graduates. Such 
data is -needed (1) to correct the errors that inevitably' result even ' 
with proper course design and (a) to update courses in the face of 
changing job requirements. . To. fill this gap-,' the Chief of Naval Education 
'and Traininq (CNETh N-34, assigned to the Training Analysis and Evaluation 
Group (TAEG) the task of 'developing a feedback method which jwpuld system- 
atically and cost effectively provide Navy schools with criticjil jnforma.- 
tion from operational units about training outcomes. To achieve this 
task, an extensive study of training' feedback methodology wa§ conducted, 
by a three-person team during the period from November 1973 to March " 
1975. - ■ ' , ' • - ^ . 

OBJECTIVES. OF THE STUDY ' • ' ' 

This a$udy was undertaken to accomplish the foliowjng objectives: 

I- T(rdetermine>'the .utility of existing training feedback methods, ' 
. pjartlcularly those which utilize ;iiexpens.iv€ mail-out procedures. 

2. . To deyelop an instrument for Nav)t trg^ining feedback, procedures 
for its administration, and procedures for utilization of the data it. 
provides. . ' ; 

3. To establish the, effectiveness and cost of the selected feedback 
method* • ' / ' " ^ 

4- To provide a manual tha\ will enable relatively dnsophisticated 
personnel in Navy schools to construct and administer the instrument and . 
to use the dat? provided. . • c 

BACKGROUND r- - 

Much of the impetus for this project stemmed, from a study by Peters 
and Chambers (1964) of Navy training .fefedback methods and requirements. 
The repoic|: identified many problems that existed in the evaluation of 
Navy training programs ranging from the general lack of awareness* of the 

^ 11 ^ ' ^ ' ' " 
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jmportwce pf traintng feedback to the absence of specific forms and 
pfaceHures for obtaining 'feedback information. A series of recomnfenda- 
"tlons.was providejl for? improvement of the Navy training feedback process, 

^ ^' BiJdftaUer (?971). noted that much the same situation^still existed 
aa/evatu^ition o^ Navy training programs as was documented in the earlier 
P^l^efs arid Chambers. (1964) report. He concluded, "the environment of 
most training programs actually prevents the accomplishment of feedback." 
To' aid, in correcting' this situation, Bildhauer developed plans, procedures 
ATtd^ri orgarrizatiflrfaT structure for an integrated Navy-wide Training 
.TejSdUa^k Subsystem |TFS)^. ; / 

^ Although 'the B^dfjauer (197f) report was prepared for tha Chief of 
Naval Personnel,' itjbecame an important working paper.in the Training . 
Appnai^l Branch (N^34) of the new CNET. Development of the TFS continued 
in this branch including a study by Lane (1972) of the use of question- 
naire* methods far obtaining field evaluation data. The conclusion of 
that report was>.thaf , questionnaires were suitable for this .purpos.e, if * 
*propej:^pr6cedures for their construction and admini strati oii, were used. 
Another result of T^ development at CNET was a second report by Bildhauer 
(1973) which documented the evolutim-of-tht/rFS from his earlier concep- 
tion afid described the developm^t and organisational structure of onie 
,lpcaT evaluation uni^t (Service School Command, Orlando). From the 
standpoint of the present study, the most critical outcprtie of CNET work 
orr the TFS was the ,^||ignment to' TAEG of the task of developing a vehicle 
to^ provide feedback format ion from the operational units to the schools. 

C'HARACTEfMSTICS OF ^ EFFECTIVE FEEDBACK METHOD ' ' ' 

t 

^^Ah optimally effective pos^ formal training feedback method wouW 
provide accurate in|prmation to the schools related to' the success or 
failure of school training for every jot) task that the trainee may be 
asked to perform, the training failures which such a feedback method 
would help to identify fall into the categories of" uridertraining (including 
mjS94ng training), 4)vertraining, and irrelevant training, 

Undertraining fts a failure' of the school to provide sufficient 
skills^ to the trairf^ to..£ndhle him to adequately perform a job task/ 
It is the most crit||cal traininh^faifure'. since, it reduces the operational 
readiness of the urjit to which thQ trainee is assigned. Overtraining 
pccurs y/hen more training is given than is needed to perform a job* task 
and training resources are 'thereby wasted. An even larger waste of 
resources occurs in irrelevant training, where skills and knowledge are 
£(cquired that liave no bearing, on. present or future job performance. 

Any post formal training feedback method that provi,des accurate 
data. on the existenpe of eacH of Jhese training failures! as well as* " 
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Identifying trainrihg successes, will nec6ssafily' be reliable and valid/ 
However, a number of additional dbsii^ble characteristics e^ist for an 
effective feedback method. • Probably the most important of ,these is low 
cost for development and a'dministPation. Someltethods, such as personal 
interviews yrith.^'Fleet personnel, become^extremely' expensive if enough < 
Visits are made to prov,ide reliable data, and mail-out questionnaires 
have .Been found to be much less expensive. (Scott, J 961 ). K is also 
desirable that a traiRing feedback miethod operates quickly. Training 
problems should, be corrected as soon as possible following their appear-"- 
artce and, therefore, the time lag to obtain indications of these problems 
must als'o be short. In addition to low cost and speed, it is^'mportant 
that the metjiod*' be' simple and easy to adfliinistei^ since any procedure ) 
.that depends on complicated analysis techniques will probably not be 
carried jout. Finally,' th'e training feedback method should require a 
^ minimum number of personnel for/its execution. Military training ^jnits 
are often understaffed and a feedback procedure that reqaires a gre,at 
amount of time by training personnel for its administration will probably 
Ij^neg'tected. ' . 

POST FORMAL TRAINING FEEDBACK TECHNIQUES . ' ' ' 

. The ultimate source of feedback information Is the performance of 
the trainee on the* job, and many ways exist to obtain such data. Probably 
no trsrining program is completely lacking in procedures for obtaining 
sdch information. However, not all methods have the capability of 
.identifying all the classes of training problems discussed above. For 
example, analysis of records of equipment failure can provide information ' 
,about^undertraining, but little or no information about overtraining and 
irrelevant training. To a. lesser extent, pift>fipiency testing shares 
this weakness. Techniques, such as questionnaires, fthi^h do have the 
potential to provide data on al.l three classes pf trainins failures, 
often fail in' the attempt, because low return rates or poor question- 
naire design reduce .the reliability of the data^ Even when usable data 
are obtained in sufficient quantities, it often is not properly analyzed 
tg identify significant trends. 
* * 

The existing techniques were the starting point in our search for 
an affective Navy post forinal training feedback method/ They are presented 
below to provide an overview of possible procedures and their relative 
adequacy.. If time and personnel are avai.l able, /some of these methods 
can be used as sources of feedback data along with the particular^long • 
questionnaire method teconmended in this report. 

pteciENCY TESTING. Objective criteria which indicate the success of 
'the trainee's performance can be measured. These include such things as 
meter readings, signal-to-noise ratios, and amplification facTtors. ' 
Along with these products of. performance, the time to .complete tasks and 
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the manner of their completion., can be observed. These various 'objective 
measurements can then be compared to standards that have been established 
as representing proper performance. .Since many aspects of military jobs , 
are not routinely performed, simulated environments can sonietimes be 
utilized to allow objective indieatiorts .of performance for these job 
behaviors. Although the resultant data provide excellent feedback 
information in ^st situations, the disadvantages are likely to outweigh 
the benefits obtained. This^ method is.trostly, tim^ consuming efhd requires 
highly skilled personnel to develop and administer the measuring instruments. 
In additidn, the equipment may not be -avail able for test i Pig since its 
use for this purpose cpuld preclude its use\in day-tq-day operations. 
For more detailed information on the reasons for avoiding performance 
-testing 'as a feedback technique see Harris and Mack^ie (1962). - 

JOB-RELEVANT KNOWLEDGE TESTING. In addition to measurej^nt' oTperformance 
in real or simulated environments, hypothetical job situations can be 
established by the use of written, questions to determine if the knowledge 
required for successful performance is present in the job incun*)enf. 
This is basically the approach of the Personnel Testing and Evaluation 
Prqgram (PTEP) being used by the" Fleet Guided Missive School. This 
procedure. is closely, related to proficiency testing and promises to have 
value for the training community, the PTEP is presently being used by 
the subsurface community for identification of specific personnel perform- 
ance weaknesses, but its potential -as a training feedback technique 
should be explored. ' ' - 

» « 

ANALYSIS'OF EXISTING RECORDSSb^ Data on equipinent failures from recprds 
such as 3M, Maintenance Data Forms,- Records of Mobilp Technical Units, 
Fleet Training Group Reports, are primarily 'hardware oriented, but to (\ 
the extent that ,they are indices of student performance C^aflures), they 
are* a source of training feedback information. Records suth as Safety 
Reports and Reports of Refresher Training can also give insight into ^ 
training problems to the extent that they reflect poor performance of ^ 
the trainee on the job. One problem with this method of obtaining^ 
information is that these records are not widely distributed. Additionally, 
the data must be verified by c^ss-checking other sources. 

VISITS: INFORMAL/UNStRUCTURED. Trai^ning personnel may visit the job ^ 
sf|e' fpr general discussion , of trainee- performance rather than to obtain 
data, on a. specif 1c training issue. This approach » suffers because of Ifhe . 
small amount of. information obtained. It may only have the advantage of 
demonstrating to Fleet units that training personnel care enough about 
supporting them to come aboard. These visits can be made more profitable 
by adding structure to the visit through the use of proper questioning 
techniques (see structured interview below).. Considering* the valae of 
the data usually collected, the cost of this method makes >t a question- 
able technique for continuing use. ^ ^ " ^ . ' 
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INFORMAL FEEDBACK. At times people in the operational environment wish 
to express their concern for, or satisfaction with, training. Such 
unsolicited information may be transmitted by letter, phone. call, or 
personal contact. Much of this information' is broad and may lack impor- 
tant details... This data should be considered as a starting point for 
further investigation,, rather than taken directly at face value. 

ROTATION OF FLEET/TRAINING PERSONNEL. Much information about the adequacy 
\ of training may be 'obtained frort neWly reporting school personnel. They 
are usually capable, well -motivated people with knowledge about, how' 
effectively school graduates perform their jobs. Information, shpu'ld be 
Obtained during skilled debriefing sessions shoHly after their arrival. 
All opinions should be backed by observation and preferably with specific 
examples. It is essential'that the information obtained in tMs way be 
evaluated in the light of the^ respondent's -experience and qualifications. 
Cross-checking with the operational unit on specific training failures 
would be important for validating the information received. The use of * 
a questionnaire which provides for ratings- of performance on all job 
ta^s^Should be used for debrief injg these personnel. 

PERFORMANCE DIARY. The Performance Diary is a running report, by the 
school graduate, of the; tasks he performs and the problems he encounters 
on the -job. It demands a high degree of cooperation between the school 
and the job incumbent and requires much time and effort on tlie part of 
the trainee. When suitable personnel have, been, fgund to keep such 
diaries, the information contained thereinjhas been useful for training- 
purposes. However, the /data obtained must'be carefully evaluated. The' 
tact that a capable technician feels a particular portion of the training 
was helpful j:o him, does' not mean that. it would help the average student 
On the other hand, if the diarist reports tttat be has difficulty, or 
cannot perfornr an operation which was taught in school, an examination 
of .that phase of instruction may be in arder. 

VISITS: STRUCTURED INTERVIEW'. The details of ttie job are 'structured in 
terms of statenfents derived from a job 'task analysis. Interview ¥orms 
tchecklists, rating scales, qu'estionrraires, etc.) are constructed which ' 
can^-be completed. 'within one to three hours of interview time. Training 
personnel visit the operational unit and administer the forms on a face- 

.-5",''^^]^ P^'^ technique has beert ^jsed satisfactorily by Bilfnski 
and Saylor (1972) and Standlee, et al., (1972) who examined the job 
performance of Navy Storekeeper graduates and electronic maintenance 
personnel, respectively; Because of. its "structure, thiS 'approach^ vf^lds " 
more information (and more reliahrle information) than -the unstructured 
interview. It is superior to mail-out procedures in. that there is" no 
problem about Veturn rate. or waiting* for replies. Also, much additional 
information can be gained about frequency of operations and maintenance 
as well as about tasks performed other than those pertaining to the 
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technical rating. Wfiile this technique 's usually to& costly in time 
and money to be used as a continuing feedback method, it is valuable for 
use in locations where .operational units are close to the' school. 

MA'IL-OUT QUESTIONNAIRE. Questionnaires have been used frequently to 
obtain training feedback data. They offer the advantage of being 'less 
costly than the personal interview method. In addition, the question^ 
naire is relatively free of interviewer bias. The format for construc- 
tion of questionnaires has varied to include open-ended questions, 
closed-ended questions, and/or rating scales. Items have been based on 
task. statements obta'ined. through job task analysis (as in this study);- 
on j6b dimensions detennined by factor analysis, as in Sieg^ and 
Federmari's (1970) study of the electronics ratings; or, as is most often 
the case, on the existing school" curriculum, procedures for administra- 
tion have also taken many different forms. The questionnaire has been 
carried by the graduate to his duty station or mailed at some specified 
time- after graduation. They have been administered to the trainee, to 
the supervisor, or to both. Also, different amounts of time have been 
permitted to lapse following graduation prior to mailing of-*he question- 
naire. Navy use of this technique has suffered from a low return rate. 
An- average return rate of . only 10 percent was reported in a recent 
survey of Navy feedback methods (TAEG^gport No. 12-1, 1973). Another 
disadvantage often mentioned has been the lack of reliability of this 
data. I.t has been argued, however, that s-atisfactory, return rates and 
useful reliable data can be obtained when appropriate procedures are 
followed in their construction and administration (Lane, 1972). 

FACTORS THAT iftPACT ON TRAINING FEEDBACK 

Many factors contribute ,to the widely varying effectiveness observed 
in training feedback efforts. For example, the very poor results /loted. 
in the recent survey of Navy feedback efforts (TAEG Report No". 12-1, 
1973) were attributed largely /to the predominant use of mail-out instru- 
ments instead of personal interviews. .Other factors that are, or may 
be^ related to the success or failure of feedback efforts, particularly 
those employing questionnaires, include the length' and format of question 
naires, ^he person or persons providing the information, the time lapse 
following graduation prior to gathering the data, the ability and 
conscientiousness of the person providing feedback data, and the effect 
of discussion or notification of the feedback effort prior to actual 
administration. Each of these factors ,is discussed below. Particular 
attention is given to the need to include each factor in an empirical 
test of feedback methods to clarify the role of the factor in training 

feedback effectiveness.- 

■* 
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PERSONALVS. MAIL-OUT CONTACT. 'Marty studies' indicate thai-well-corlstructed 
and carefully administered mail -out ins.trumentfs can provide accurate and' 
reliable data. These include studies- (Bougler, 1970;'Scott, 1961) ,in 
which no differen-ces were found between data obtained from mail-out 
interviews and personal interviews. However, mail -out procedures' were 
if !![ X ^•'^^TAir^n '"^""^ survey of training feedback practices • 

>n the Navy (TAEG Report No. 12-1, 1973) because of the low return rate ' 
and because the data were often judged to be unreliable. Prior to the • 
selection and , development of a feedback method for Navy-wide use, it was 
very important to 'resolve these contradictory positions about the effec- ' 
':nrnf^LSri"'-^>r"°"^ P'^oc^dures. Since a cqjnparison of mgil-out interviews 
and personal interviews had not been done in the area of military training, 
an experiment was needed to compare these procedures in a Navy training' 
setting, if mail-out procedures were found adequate for the job, the ' 
sav.ings for the training conminity would be huge. 

LENGTH AND FORMAT OF FEEDBACK INSTRUMENTS. Long questionnaires provide " 
the- opportunity' for a more comprehensive evaluation of training than 
sho»;i; questionnaires. However, long questionnaires are more apt t6 
produce fatigue and boredom in the respondent. Thus, the benefits of 
length. may be cancelled because of reduced^ccuracy of the data or 
because very few questionnaires are returned. For these reasons, short 
questionnaires might provid^hetter data- ±han long questionnaires despite ' 
their limitations. To obtaijf^etter knowledge of the relationship 
between questionnaire length and effectiveness, .it was essential to 
•include both long and short questionnaires in the present experiment 
comparing mail-out intervieW^ith personal interviews. Data from the 
long questionnaires would also provide the opportunity to assess the 
accuracy of questionnaire data 'from different item locations. Presumably,- 
Items appearing early would provide better" data and the number of item's 
into the questionnaire at which data become less^ccurate would provide' ' 
an Indication of the maximum length, for questionnaires. ^ 

"Resp^ding to mail-out questionnaires is closely related to the 
kind of behavior required fcfr paper and pencil examinations. Some of , 
the poor fesults observed in Navy use of feedback questionnaires coald 
be related -to an aversion for such examination-Vfke tasks. Bilinski and ' 
Saylor (1972) used a novel format for feedback instruments in an assess- 
ment 'of training for-the Navy Storekeeper rating. Thfey conducted 
structured! interviews in which cards, with printed task statements, were 
sort^intp one of several different categories related to leVels of ^ 
perfohijancfe of a trainee on the task. This successful format' and proce- 
dure for^terviews appeared to be adaptable for mail-out instruments. ' 
Because of^s simplicity and novelty, it might provide a higlrer return 
rate and tno^\^6li able and more valid-data. than more traditional > formats, 
The possibleWvantages of this technique argued for its inclusion in 
the study compaidng feedback methods. 
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TRAINEE VS-. SUPERVISOR [ADMINISTRATION PROCEDURE; PVobably the most • ^ 
.important question related to administration of any feedback' method is 
that, ofi who should provide the ratings of ^trainee performance.. The 
trainee (|as a, direct and recent knowledge of school training, and also ' 
has a diftect,' though limited, acquaintance with the^quirements of his 
rtew jolii* On the other hand, the. supervisor has a much more .complete and 
accurate '*C9ncept ion of-^dequate job gerformance^but litfte knowledge of 
current training at^the school. * Thus both^the trainee and supervisor 
have speci|il perspectives that; in theory, jnake jjiput from both essential 
for a complete training feedbafk data base. However, even if their data 
were identncal, it woOld be useful to share the workload between Jerainees 
and supervisors. If both 'tr^tinees and supervisor? are to be conflicted, 
the ajJditi'onal. question arises as to whether both members of a tr^ineer 
supervispjf|,paiV should be interrogated or only one member from each , 
.pair. ^ Jgxperiment investigating different" methods could aliso be used r 
to compare the accuracy and reliability of data from these different 
groups of arespondents. / 

* I' • 
TIME LAPSE .PRIOR TO ADMINISTRATION, Another factor impacting on training 
feedback ii^the lapse 6f time following graduation prior to mailing .a 
questionnaire or conducting an interview. More time on the job will 
allow a bep^r conception rof job requirements to develop in the trainee, 
but it wiWallso-lead to more foicgetting of school training problems, , 
particula]^|r'iIthose problems that are ..corrected by on-tKe-job training.' 
A similar.Mn^^^ and disadvajitage would acCrue to supervisors as elapsed 
time betwe^b^^'aduation and evaluation increases. They would have more 
'time *o oS^Mve the ;tra'lnee on the job but more possibility would exist 
for confoun^g of on-the-job' and school training. In a Study of feedback 
methods, tl\^flata from respondent groups made up of persons with ctifferent 
times from .g^duat ion could be compared to help provide an answer to the . 
complex qud^on regarding the optimal time' from graduation to^'wait 
prior to oHSSining training feedback. " « * 

EXPECTATION Oi^ ^SET", tSgjRESPOND. Repreijentations of the future such as 
goals and exjssctations play an extremely important role in human behavior. 
A person wh& -receives notification that he will 'be receiving a feedback • 
questionnaire^ will probably be more receptive to it when it arrives than 
the person #ib is surprised by its appearance. However, such expectations' 
might work iKi reduce the accuracy and care of rjesponding, at least for 
some responSlAt groups.. A test of the effects of prior notification by 
"expectatiortpetters" was needed. ^ * . * 

CLASS STANDING OF THE GRADUATE. Training is probably experienced very-/ 
differently persons who graduate at. the top, middle and bottom of 
their classed! In addition, the more conscientious and able student may 
.be a more CQ^&ientious and accurate provider of 'feedback data., Good ^ 
reasons thuslexist for anal^yzipg feedback data separately for these 
groups and an 6xaminat'ion of these possible differences was included in 
the study off feedback methods. ' ^ 

1- 18 • '< 



TAE6 Report No. 19 ' 

.;*■■' ; . ; " . 

. ^ ■ ■ • SECTIOH II ' • . 

■ • '. TECHNICAL APPROACH* ' ' ' ' - 

A number 'of issues critical to- the selection and developmenf^f a 
Navy training feedback method required emp.irical data for resolution. 
Fror this reason,, an investigation was conducted in, which different 
feedback methods were applied in an actual Navy schooT;/ job setting. 
Specific training problems were already knpwn to exist. at 'the chosen . 
test locatiort (the Radioman "A" School) and this- allowed comparison of 
the different instruments and procedures to determine the effectiveness 
of each for such problem identification. Of more importance, it provided 
the answer to the question o^ whether or not any of the methods could 
provide this information from the operational units to school s^out • 
training problems. ... ■ ^ 

VARIABLES INVEST I GAT£D IN THE STUDY . ' 



Each of the factors impacting on feedback effectiveness described?'" 
in the previous section' became a variable that was iny^tigated in th~e 
study. In addition, a variable based on where the gradjuate was stationed 
^. was included, since this was closely ^elated to the pro|>lemS known to 
-exist in'the training curriculum for Radioman. "A" School. ' 

"• ' . ' , 

The following variables were investigated a^jd are discussed below. 
'Variables related to the instruments included: 

• Type of instrument ' • * ^ 

Normal vs. reversed item order 

Variables related to characteristics of personnel receiving instruments 
included: 

^ ' : ' ' 

Supervisor or trainee responder^J; 

Joint vs. single administration ^ 
' . Class standing of graduate / 
# . Time from graduation 

Expectation vs. non-expectatipn , 

Ship vs. .shore duty station , / ^ . 

The major variable under investigation was the type of feedback 
instrument. Three instruments use4 mail-out procedures ta obtain data 
and one" used a structured personal interview. One mail-out instrument 
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• was a long questionnaire that required ratings of tas'k frequency, task 
crnicalityj and trainee jserfontiance on 134 specific job tasks performed 
by Ratdiomen. Another wasf a-short questionnaire that required the same • 
ratings on 15 general job tasks which subsumed the 'specific job tasks of 
the long questionnaire. Jhe final mail -out instrument was a card-sort 
^ procedure where 134 cards,., with the specific job tasks, wer-e sorted into 
different 'Stacks related to the -f req.uency of , task performance and the 
. level of trainee performance on -the task. The structured Interview used 
the long questionnaire, but it was administered in a face-to-face situation 
with the interviewed writing down the ratings as they were" told to him 
by ^the person being interviewed. 

> ' * ' • • 

■ A second variable in the study also' involved the type of instrument 
used. This was a two-level variable defi'ned .by'whether the- items on the 
instrument were in normal or reversed order. This variable was includedr 
to obtain information about possible differences in ratings of an-item ^ 
depending upon the position^ of the item in the questionnaire. It was 
completely crossed" with the three-level factorjof mail-out instrument 
- -type. This means that one-half ofithe questionnaires of each type had 
reversed item order and one-half hid normal itero-' order. It also means 
that one-third -of the reversed-itefn-order instruments were long question- 
naires, one-third were short questionnaires, and one-third were card . ^ 
sort instruments. Fina.lly, this complete "crossing" of these two factors 
implies that the same propositions (one- third of each q.iiestionnaire type) 
held for the instruments with normal -item-order. 

The oth^r six variables included in the investigation are all . 
related to characteristics of the personnel who were mailed the feedback - 
insthjiijents. The most important of these variables involved whether thV 
respondent was a supervisor or trainee. Another two-level variable 
(referred to as joint vs. single adminis trail on) was determined by 
whether the "instrument was mailed to only one member of the supervisor- . 
trainee pair or to both members of the pair. A three-level variable was 
defined by the class standing of the graduate. Class graduation rosters i 
' divided into thirds on the basis of class standing to allow separate 

investigation of feedback accuracy for groups at, the top, middle and 
bottom of their classes. A four-level variable was determ.ined by the 
time lapse from graduation prior to administration of the instrument. . 
•The groups were 4 to 6 months time lapse, 7 Jo 9 months, "10 to"l2 months, 
and greater than 12 months. A two-level variable was determined by 
whether or not the person had received an "expectation letter" prior" to 
\ the questionnaire, informing. him of its imminent arrival. The final 
• tworlevel variable was .related to whether the graduate was assigned to a 
ship or to a shore duty station's 
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. Some of. these variables,' such as'dufe/ station of the aradnstP *;,I!h 
- tirne from graduation, were not under tKntrS of thrinJ^s'tiaatorJ 

WhefeSefJoL^bfe assi gned- randomly to trainees and supervisors, 

wnerever possible, equal numbers o-f each level of a variable were inrinHpH 

. .t:»,-S lifferent man-out instruments were assigned to pirsonnel • 
pS ?e l*,™?'^ I^H)*'*'""? P^"""-"^' aboard sSff ?""? ls . 

• s?Jg,eS ^S^' ^""'^l ^V^ater ^^tli'Ali]^^^;, 

fr.il establish possible- differences among the above factors the > ' ' 

[n ISt 'r^'"J-''"J^l"- "'^^ time uriHl retS^n'of the ^ 

instrument, percentage of instruments returned, average ratinas fnr ?nh 
task Items, and finally, a measure of ques'tionnai>f ??mp'et ac?JraS 
force^^lTttli: '''''''' ta^kVequenc^ f^ K^S; ??eq^^^Scl^^ 

SELECTION OF A SCHOOL/JOB SEHJNG EOR THE STUDY " . 

e^^hihi-Jp?^ essential to^identi/y a training program which actually - 
nJnhw / substantial number of each of the thfee classes of traininq' ' 

S^deJ to is;abnrh''j;"3l.r!:^''^'"^'"9* irrelevant training)! ' ^ 

™ • effectiveness of any feedback method, m to make 

valid comparisons between different methods and different qrouDs of ' • 
respondents. Arv additional .requirement for the progrim was ?hat a larae 
number of school graduates be available to obtain rlliabTecomoar i sons ^ 
among a number of instruments, groups and conditions. ^o-^P^^isorvs, 

The Radioman. "A" School most nearly met the criteria stated -im fha 
paragraph above and was selected as the teJ^ bed Sr ^h^dSpm^nt "o? 
2.TcSE!k/''^'''"^ instr^^nts and procedures. This school warSee^d • 
most suitable since at the'^time of initial contact' a major cu??iculur 
pmi-ou°sl5'' ^h^np'r" ^°?Pl?t^^>ased.on a task ana^ysi's Lde tw "Xs 
fTf^c ihl?* The -new curriculum included a number of important ne» iob 
tasks that were not includ|fl in the old, and thereby corcected manv 
iSe'ma^nr ^[.""dBrtraininq^hich existed in the old'cu ri^ m."' Lver 
the major difference between the old and nfew curricula was that seoaratp' 

T\l1e^:ra1e';r1k^•^o'.^?'•^'^^'°^^' '^^^^'^"^ and ?ors!;ore.\'L\ 1 ; Smen; 
jne separate tracking eliminated a great deal of overtrainina anH ' 

■'^^o^^t |MRbo.rd.coLn1cat1ons tSsfS^fo^'shore- ■ 
Mseo pereonnel. This also eliminated .the waste of training resources 
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that existed for shipboard personnel who were/faeing ^ught a great deal 
.about sKore commun feat ions that they might never use or could obtain 
later in their"- careers as the need arose. 

Although the new Radioman "A" School curriculum had been^dev^lapfd,' 
it had not yet beeti' implemented and the fisting curriculum was still 
producing ^ipp.raximateTj^ 50 graduates per week from the Bainbridge and 
'San Diego schools. This large number of .graduates , plus the documented 
examples of training problems, made this an ideal setting for an empirical 
comparison of feedback methods. In addition; school personnel and the 
Training Program Coordinator for the school expressed a willingness' to 
cooperate and aid in this effort. : , - ' . 

IDENTIFICATION AND LOCATION OF SCHOOL GRAbUATES AND SUPERVISORS\ 

, ^ • / : ' , " - . 

Prior to administering feedback instruments, it was essential to 
have accurate data on the location of Racjipman "A" School graduates and 
to have the' names of their supervisors, . One reason for this was that 
only one mail -out instrument would be mailed to a supervisor regardless 
of the, number of graduates he supervised. It was feared that mailing 
more than, one instrument to a supervisor would be burdensome and'cause 
supervisor resistance to the study which in turn would. reduce the return 
rate of the instruments. ,Also; acdurate pinpointing of graduate locations 
would insure that return rate statistics wpuld be less influenced h}/ 
delayed or Ipst questionnaires,-^ Finally, one of the key variables of 
the study was the ship or^shore location of the graduate, and accurate . 
identification of his whereabouts wa*s. essential for correct data analysis 
on this ship/shore duty station factor. 

To obtain these data^ Tetters were prepared and mailed to the 
different operational units receiving' Radioman "A" School graduates. 
Communication Officers were asked to confirm whetf^r the trainee was 
stationed at the unit and to add to the li^t t^e rrafties of any additional 
recent graduates, * Finally, they were asked to add the name* of the 
supervisor who was most familiar With the graduate's performance from 
,the date of his arrival. . - , . 

Permission to contact operational unit? was obtained from the 
Atlantic and Pacific Fleets and from the Nayal Telecommunications Command, 
At the saijie time, permission was obtained to mail the feedback instruments 
directly to the trainee amjj to thie. supervisor. Such direct contact 
reduced handling of materi^^l^- by ^^ommand personnel. With the large ' 
number of persons to be contacted in 'the study, such handling could have 
become burdensome and. a potential source of non-cooperation. 
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IMSTRUMENTS. • ... 

. Thr*ee , vers ions of the mail-out questionnaire were compared in this"" 
'study. These were (l)"ashort questionnaire wi^h items based-on general i 
job tasks including several open-ended question* tq get, at specific 
training problems, (2) a long questionnaire based on specific job tasks, 
including' pne open-ended question, and {3\ a. card sort procedure using 
. the same task list *and open-endfed question as were ^sed for the long 
.q\jestipnnaire. Two different versions were prepajed for each of these 
instrume,nts, one for the tirainees and one for the supervisors. In 
^addition, versions of each were prepared" with normal and reversed item* 
order as a means of exploiting changes in the accuracy of responding as a 
function of the length* of the instrument. Fatigue or boredom effects 
would show, up as differences in average ratings for the same. item when 
If was encountered early in one version aild later in another. 

All three" mail -out instruments were designed to provide information 
about undertraining, overtraining, training relevancy, and missing^ 
training. The loifg questionnaire and .card sort instrument provided 
infohnatjon on the fir^t three training problems by obtaining ratings of 
specific job tasfe that were listed in the instrument. Missing training 
objectives Were to be ident^ied % the open-ended questig^n. "Specific 
tasks were not lilsted on the short questionnaire, only general ones. 
,Seven open-ended questions were added to get at undertraining, -over-* 
training, and training relevancy for specific job tasks as well as to 
get at. missing training objectives. 

SHORT QUESTIONNAIRE. The 15 general ta.5k. statements used, in the short 
questionnaire are presented in appendix A. At a very general 16vel' 
these describe the work performed by Radiomen at Fleet and shore units. 
These items were, typed on two pages of the form presented .in appendix C. 
This basic form (with minor changes-) was used on both the long and short 
questionnaires and provided ihe three scales of Frequency of Task, 
Cr4ticality of Task, and Performance of Task Upon Arrival From School;" 
space for eight'task descriptions; and a brief set of. instructionsv The 
respondent indicated' his rating on these scales by circling a number 
correspondijig-tp one of the" five available alternatives for each. Seven 
open-ended questions (appendix D) were added at the end of. the short 
uestionri&ire. The initial page of the questionnaire was a cover letter 
appendix E). If was followed by instructions (appendix F), a biographical 
data sheet (appendix G), ^ep the general task pages and, finally, the 
open-6nded' questions. Appendixes D throug°h .G are from the trainee " 
version of the short questionnaire. The supervisor version jof^hese 
forms differed only in minor changes of wording.. Time to complete 'the 
shont 'questionnaire was about 13 minutes.- ■ ' ^ 
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LONG QUESTlONN/VIRE- The long questionnaire utilized the 134 specific -7 
job ta^ks presented in appendix B. Most of these w^re taken from both 
the old and new curricula; Thos^ rejated to radiot;^egraph operation 
' came from the job task analysis, but are'not included in either the old 
or new curricula. *The; questions'. were ]jrin ted oh 20 pages of the sam^ 
basic rating scale form (appendix C) u^sed in the short que^ionnairfe, 
^ind made up the bulk of the document • the ^me -cover letter, instructions 
and biographical <Iata sheet as those used on the short questionnaire 
preceded these tasks and rating scales.' The finaT page of the long 
questionnaire was a .s-ingle open-ended question (appendix H) designed to 
identify new areas of instruction which should he added to the curriculum 
of Radioman "A" School. 'Supervisor^ versions of the long questionnaire 
presented only minor modifications of the forms in 'the appendixes. The 
tjine to icomploto tho.loff S questionnaire averaged approximately one hour. 

> .CARD SORT PROCEDURE. The card sort procedure' used the same 134 it-ems as 
the long questionnaire^ These' were printed on small cards (2.25 in. x 4 
in.) with one task statement per card. A "placemat" contained six i 
rectangles labeled with the categories into which the 134 cards were to 
' b? sorted and also included the instructions for this sorting.'^ The 
''^H)lacemat" is presented in appendix I. Labeled rubber bands jwere. enclosed 
to be used following sorting of !the cards to make six separate packets . 
that were to be returned along with the same biographical delta, sheet 
(^appendix G) and open-ended question (appendix H) that were used wl^h 
the long questionnaire. , A Supervisor "placemat"yWUh minor changes was 
also usedv Time to complete this card sort ppa«dure aver^iged about. 30 
minutes* . » ' . . 

STRUCTURED INTERVIEW. In addition to the mail -out feedback procedures, 
face-to-face strflctured interviews were conductec^j^ A long questionnaire 
was used for this purpose with separate, copies for the interviewer and 
interviewee. The person being interviewed was asked to read the task 
statement aloud then tell his ratings on the three scales to the inter- 
^ viewer who marked these on his copy of the long questionnaire,' After 
'all tasks had been considered, those j^/Mch received "inadequate" Perform- 
ance of Task Upon Arrival From School ratings were reviewed and specific 
questions about training for these tasks ^asked. Answers were recorded 
by the interviewer. Suggestions for Additional .material to be included ' 
in the curriculum were also requested and recorded. This procedure 
guaranteed that consideration would be^given to each item and produced a . 
large amqunt of discussion relevant to training 'for the different 'ta^.^ 
Average tTmr^or these interviews was one liour and twenty minutes. 

SUBJECTS • ' " " ' ' 

Instruments were mailed to 996 trainees .and . 590 superv'isors and 
. strijctured irjterviews were administered to an additional 59. trainees and 
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37 supervisors. The names ol'.l;hese personnel were o^btained from verified 
lists of graduates returned from operational units. The trainees were 
graduates of the unmodified Radioman "A" School cXirriculum during the 
period May 1973 to July 1974. Almost Without exdeption-, these graduates 
were early in their first enlistment with most going di rectify from ' . 
recruit training »to "A" Schbol. Supervisors, were usually Pirst Class or 

„^ief Petty Offipfers with 10 or more years of service. "The different . 

•i.n.tmbers ^f trafnees'and supervisors, resulted fropi the fact that a super- • 
yt^or often supervised more than one trainee. . ' 



More of these personne^ werje stationed at shore installations (885) 
than aboard ships (701) and the difference reflects tie 'fact -that a 
m^ority of Radioman "A" School graduates receive their first, assignment 
at' shore stations. Two or "more trainees are more fipt to have the same 
supervisor if the trainee is at a shore ins'tallati on than if he is 
aboard ship.* This difference can be seen in" table 1 'which presents a 
breakdown of subjects in the study by factors of supervisor ys. trainee*, 
feedback method, and duty station. • ' . -.^ ' , . 



TABLE i.®"" NUMBER OF TRAINEES AND SUPERVISORS AT- DIFFERENT 

DUTY STATIONS RECEIVING DIFFERENT INSTRUMENTS 
- , \' 





•Long 
, Questionnaire 


Short, 
Questionna'ire. 


Card 
Sort 


« 


Structured . 
Interview 


Trainees: 






} ' ' ' 








Ship 




'136 ' 


144 . 


135 






Shore 




192 


m 


193" 




41 • ■ 


Supervis9rs: 














Ship 




"96 


'■ 101. 


89 




19- • 


Shore • ^ . ' 




105 


99 




18 








■ \^ 

• ' '\ 
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The fact that many trainees shared a supervisor and only one instru- 
ment was mailed ^o a supervisor r^ulted in another imbalance in the • 
numbers of subjects in the Joint vs. single administration conditions. 
Three hundred and fourteen supervisors were selected randomly andassigned 
tp the joirrt efdmi^ni strati on conditiftn. This made a total of 622 trainees 
and supervisors receiving joint administration* The remaining 279 
supervisors of recent graduate^ received instruments but recent graduates 
they slipervfsed not. The remaining 685 available trainees also 
received instruments but their supervisors uid not (unless the supervisor 
received a form tj) complete on another trainee). This made a total of 
622 persons in the jo int-^admini strati on condition compared to the 964 
with single admir^stratijjn. 

Approximately equal numbers of subjects were assigned to different 
levels of the factors of instrument Type, Clas^ Standing, and Item ' 
Order. Differences did exist in the^ number, of subjects ip the different 
Time Lapse from '^aduation groups. These differences reflected normal 
variations in.grauiuate output during the year and also the inclusion of 
graduates from a ^'ve-month period in the greater-than-12-months time- 
lapse group. The^nly other imbalance that appeared in nun±)ers for 
different level s^pf a factor was tn the Expectation Letter vs. No Expects- 
tion Lettfer grou^. This resulted because not all data requests from 
operational unit? had returned before questionnaire mailing began, Late^ 
arriving names |rainees and their supervisors were mailed question-: 
naires> but to pfcejvent further dejays in their mailing, none of these 
personnel received! *the prior expectation letter. For thi-s reason, the 
number of questi^naires preceded by "expectation letters" (758) was 
smaller than the^umber ^nailed without these letters (828). 

The imbalances in numbers of Subjects that existed in many conditions 
were -not large enough to reduce the reliability of data from smaller 
groups'. However, data from the^e groups required careful analysis and 
interpretation ta^^avoid confounding of effects associated with one 
variable with tli^'effects associated with another. 

PROCEDURE I * ' . 

Four to six/weeks prior to mailing of the .instruments, approximately 
one-half of the trainees aad supervisors were mailed a letter announcing 
the arrival of t|ie feedback, materials and requesting cJonsi deration of * 
training problems in Radioman "A" School. In the joint administration 
condition, where both trainee and supervisor were to be mailed instruments, 
both persons received the "expgctat^ion letter'* or neither did. The bulk 
of the feedback 'instruments was mailed November 17 through November 19, 
1974 with the remainder being mailed over the next month as additional 
verified data oh trainee location and supervisors were returned from 
operational unfts. -.Each instrument was accompanied by a self-addressed 
return envelope. * - 
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Follow-up procedures were initiated at seven and at nine weeks 
after mailing of the evaluation m&teriails. The first follow-up to non- 
respondents was a simple .reminder letter requesting the return of the 
questionnaire. The second requested that they indicate their reasons * 
Sot not returning the jnaterials and askedifor suggestions for improve- 
ment in future evaluation effort?. A f«low-upvletter was also mailed ? 
to those persons who had completed and-wtumed /the questionnaire asking 
them to reUte any problems they may have had coippleting the materials 
and: to suggest improvements for- future feedback efforts. 

-As instruments were, returned,, the rating scale data were inmediatel' 
punched on cards for analysis. A seven-digit identification code was 
included on these cards that designated the status of the respondent on 
the variables of Trainee vs. Supervisor, Joint vs. Single Administration*, 
Questionnaire type, and the remaining five variables studied. When it 
appeared that no m6re instruments were forthcoming, statistics were 
calculated including return^ rates for the various conditions and sub- 
conditions, averages and standard -deviations of rattfig scale data, and 
similar statistics jDn time to return and numbers of comments. • 



/ 



TAEG Report Jlo. 19 



THIS PAGE IffTENtlONALLY LEFT BLANK 



28 



3?2 



ERIC 



TAEG Report No, 19 




SECTION III 



RESULTS 



Results are presented for the major dependent variables of instrument 
. retulm rate, responses on the rating scales, responses to open-ended 
questions, and the results of follow-up efforts that took place after 
the main comparison of feedback methods. Descriptive statistics are 
tabulated and the statistically significant differences are described in 
the text. The decision rule selected fop statistical significance was 
that a result should occur hy chance only five times or less .out of 100 
(P<.05). , T 

INSTRUMENT RETURN STATISTICS / j 

The instrument return statistics include ttie overall .return rate 
for instruronts, the time from mailing untvl questionnaire return, and 
the percentage of persons receiving reminde^-^tters who responded 
following the reminder letted. These instrument return statistics are 
described below for each of the factors included In th§ study. The 
order of presentation of factors i^ generally related to the magnitude 
and importance of tfie observed differences. 

RESPONDENTS. Statistics on* instrument return for trainees and supervisors 
are^ presented in the top part of table 2. The return rate for supervisors 
^was a]most double that of trainees and the difference wds hijAly significant 
. ' (z = 5»2, p<.001}. The percentage of reminder letters producing returned 
instruments was also much higher for supervisors than trainees and 
highly significant (z = 6.5, p<.001). The difference in average time 
from mailing to return was not significant for these two groups. 

INSTRUMENTS. The data for instruments are presented in the middle of 
table 2. There was 1 significantly -higher retum rate for the short 
questionnaire than for either the long questionnaire or card sort proce- 
dure (z = 3.7 for short vs, long; z = 3A for short, vs. card sort; p<.001 
for both). The smalV difference between the long questionnaire and the 
card sort instrument was not significant. Differences among the three 
instruments for the percentage of. reminder letters producing returned 
questionnaires also were not significant. However, the longer average 
time to return for the long form than the short form was significant 
(t = 3.2, >df = 460, p<.01), as was* the difference in time to return 

• between the^long form and the card sort instrument (t = 2.0, df = 392, - 

. p<.05). r 

Instrument return data on the combined factors* of Instrument Type 
and Respondent are presented in the bottom part of table 2. For trainees, 
the return rate for the short form was significantly higher thain for--the 
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TABLE 2. NUMBER OF INSTRUMENTS MAILED AND RETURNED AND AVERAGE 
TIME TO RETURN FOR DIFFERENT RESPONDENT GROUPS m 
INSTRUMENTS ' • ■ 



Instrument 
or 

Group 



A, • / ^"1^^^^ ' 



* ^ 

Respondent 


Trainee 

Supervisor ' 


• 996 
Ron 


310 


31 1 
59.3 


66 


21 .6 


1 

1 

35.1 j 








Short Form 


. 546 


266. 


48.7 


38 


11.9 


30.7 I 


Instrument 


L6ng Form 


524 


196 


'37.4 


- 43- 


n.6 


38.8 j 




Card Sort 


516 


193. 

; 


38.4 


28 


8.r 


33.2 J. 




Short- Trainee 




340 


1 

136 


40.0 




' 1 

8.5 


} 

31.0 , ■ 




Long-Trainee 


328 


8| 


25.2 


14 


5,4 


j 

36.2 ! 


Respondent' 

1 


Card Sort- Trainee 


328 


91 


27.7 


10 


4.0 


i 

32.2 j- 


! X 

1 


Short- Supervisor 


206 


130 


63.1 


19 


20.0 


30.4 


j Instrument 


Long-Supervisor 


•196 


113 


57.7 


29 


25.9 


4r.3 1 ■ 


! 

t 

I 

1 

i 


Card Sort- 
Supervisor 


188 : 


107 


56.9 


18 


18.2 ; 


34.0 


} 


« 


■1 


. -1 


. 1 
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long form or tfie card sort ( z = 4.0 for short vs. long; z = 3.3 for* 
short vs. card sort; p<.001 for both). In addition, for trainees the 
percentage of short questionnaires retuhied as- a result of reminder 
letters was significantly higher than the corresponding percentage for 
the card sort instrument (z = 2.0, p<.05). For supervisors, -the corre- 
sponding, differences between instruments were considerably smaller and* 
not statistically significant. 

TlftE LAPSE PRIOR TO EVALUATION. Dgta for groups with different tin.es 
frqn graduation prior to evaluation are presented in the top half of 
table 3. The 4 to 6 months group yielded a lower return rate than each 
of the two longest periods (z = 2.6 for the 4 to 6 vs. 10 to 12 groups"; 
J = 2.5. for. the 4-6 vs. greater than 12 groups; p<.05). The other 
return rate differences among these groups were not significant. None 
Of the differences in time to return and percentage of instruments 
returned following the reminder letters were significant for these 
groups. ^ 

ADMINISTRATION PROCEDURE. InstrU^nt return statistics are presented at 
the bottom of^tabl-e 3 for the^faotor of Ooint vs. Single Administration 
of the instruments. Unequal numbers of trainees and supervisors in the 
single administration condition required the breakdown of these results 
by the Respondent Group factor. Although a higher return rate iVi favor 
of ipmt administration appeared for both trainees' and supervisors, it - 
was not significant (z = 1.6, p<.ll). No significant differences 
occurred between joint and single administration on the other return 
statistics. ^ • 

CLASS STANDING. Instrument return statistics are presented in table 4 
for >the. different Class Standing groups with separate' data presented for 
trainees and supervisors. The significant differences for trainees were 
on thp percent of instruments returned and on the percent of successful 
reminder letters. Trainees in the bottom third of their graduating 
classes returned significantly fewer instruments than trainees in the 
upper and middle third of their classes (z = 3.3 for lower 1/3 vs. upper 
1/3; 2 = 2.7 for lower 1/3 vs. middle 1/3; p <.01 for Vothh Trainees 
in the lower third cf their classes also accounted' for a lower percentage 
of reminder letters producing return than- was found for trainee^ in the 
upper third of their classes (z = 2.3, p<.05). Supervisors to6k signifi 
cantly longer to return instruments for the lower third of graduates 
than the other groups (t = 2.74, df = 238 p<.OT for lower 1/3 vs. upper 
1/3; t =-2.23, df = 22U p<.05 for lower 1/3 vs. middle 1/3). 



31 



TAE6 Report No. 19- 



TABLE'3. NUMBER OF INSTRUMENTS MAILED AND RETURNED AND. AVERAGE 

TIME TO RETURN FOR DIFFERENT TIME FROM GRADUATmM GROUPS 
•AND ADMINISTRATION PROCEDURES 




Time 
Lapse' 
Prior 
To 

Evaluation 
(Months) 


4-6 

7-9 • 
10-12 

More than' 12 

1 


340 120 * 

408 168 ; 
279 127 
559 -245 


1 ' ' ' ' 
35.3 , 19 7.9 ; 34. l' 

41.2 ! 32 :ir.8 ! 35.8 

45.5 hl.l • 33.4 

! ..1 • 1 
43.8 i 39 111.0 . 34.7 

i ' 


Administra- , Joint - Trainee 
t^ve 

Procedure 1 Single - Trainee 

! 
f 

1 Joint-Supervisor. 
^ ! Single-Supervisor 

i 


311 ' 106 

f 

685 '204 'j 
311 : 192 Ij 

279 : 158 :! 

1 


34.1 .1 16 

1 

29.8 27 
61.7 38 • 
56.6 ' 28 

• 

\ 

i 


V.Z ; 31.4 
5.3 ' 32.6 
24.2 ; 35.5 

t ' [ » 

18.8 • 33.7' 



32 



36 



■ TAE« Report No.' 19. ' 

TABLE 4. NUMBER OF. INSTRUMENTS MAILED AND RETURNED 'AND AVERAGE 
. . TIME. TO.RETURN FOR DIFFERENT CLASS STANDING GROUPS 




Trainee 

Class 

Standing 


Upper 1/3* " 
, Mfadle 1/3 . • 


^ 

! 320 
. 333 


/ 

115 

112 


35.9 
33.6 


18 
16 


8.1 
6.8 


/ 

f ( 

1 

35.8 
31.9 \ 




Lower 1/3 


343 


83 


24.2 


9 


3.3 


28.1 


• 

Supervisor 

Class 

{Standing 

i 

1 


Upper 1/3 
Middle 1/3 


^ 176 1 


127- 
110 


59.9' 
62.5 


22 
14 


20.6 
17.5 


28.2 
30.5 j 


Lower 1/3 j 

1 

, L 


202^ 

f 


113 

1 


55.9 

\ 

{ * 


31 

» 

1 


25.8 

• j 


40.5 i 
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OTHER FACTORS. Instrument return statistics for the factors of Duty 
Mo!r°!?'*J*®1l'.2?®'"' ^"^ Expectation Group are presented in table S:' 
None of the differences between, ship and shore duty station were signifi- 

wUh m™a?^?L*^'H^'P'"^'"* variables. This was. afso tVue for instruments 
mtn normal item order as. opposed to reversed item order. Significantly 

1'!!^""^"^.^^^ returned, however, for the group receiving "expecta- 
^J^J^^JP questionnaires than for the group that did not ' 

^nP^fc^^"' letters- 2.0. p<.05). When data f?om trainees anS • . 
'^ffrlZ 1 "1.1^ considered separately for the "expectation letter" 
= l!8l^'p< 05) " ""^^^ Clifference for trainees was significant (z 

RATING SCALE DATA ' ■ 

'nf fr-^niL^"^^ ""^^^"^ ^'^i^l P'"^"^ "^^"5 for identification 

?L r successes and failures for different tasks, they are also 

Jn^truLnJc^ ^""^^'S^ instrument effectiveness in the comparison of 
.instruments and respondent groups. For a.ll 'ra.ting scales the scale ' 

of tasks'".^nH L'nH n"%*° "ig^ .^^^quency of tasks, high critical ity 

on tSfreip^ecti^e seal's^ '''' indicated by high numbers- / 

^warSS^Lf?fLr.T- jf'^ -source of Variance in all comparisons 

was among the items themselves. This was particularly true on the 
frequency of Task scale where overall means ranged from 1.25 (item No. 
^24 of appendix B) to 4.6 (item No. 6 of appendixA). Since only a 

tlZfnfl Sc ^°^!: '^"^^^ ^^i^*^^ °" ftVe-poi-nt scales, this 

Tn 5!*. ^'^^ ""^ ratings, averaged over all subjects, indicated 
- an extremely accurate and consistent pattern of responding by nearly all 
S?iEornr"nlp5° h''^"'!;-^ questionnaires. Even mofe ext?em^ average 
Sp "2 nSf 3 wjen ship and shore re<:.nts were considered separately. 
Sr^beTlow 1 5 ' ^^e items showed average Frequency ratings above 4 5 

in..Jl!^toS''!n^Jj; ^D- consistency of ratings were also dramatically 
Illustrated in the high correlations found when average item ratings 
were correlated between different instruments, betwee^ res^ndeni qrouos 
and even between the ratings on the different scdlesle'g fPJeSufeSJ?^ ' . 
Th^Sp ''J-"^'*^^- Co^'-^lations between average item ratings for the 
three rating scales are presented in table 6. It can be seen that the5e « 

'J';?' between scales on the short for^. -^re lu lkTr'y 1 ?0. j 
A though these correlations ^vere also high or*' the long form of the 
Jnr^f ^I-^l' ^lu^ "t^ significantly lower than the corresponding 



TABLE 5. 



TAEG -Report No. 

NUMBER OF INSTRUMENTS MAILED AND RETURNED AND AVERAGE 

^^^^^ FOR DIFFERENT DUTY STATIONS, ITEM ORDER» 

AND EXPECTATION GROUPS ' . ' ' 




1 

lOuty 
Station - 

V 

_ 


Sho rp i 
ShiD / 


ooo 
/U 1 


1* 

i ' 
o/f3 


/' ^ ^ ^ 

/I O 1 

42 ••1 ^ 
40.9 




7' 
56 


'*/ ^ > ^ 

! r 
9.4 

11.9 - 


32.3 
38.2 

r 


Item- 
Order 


Normal Order 
Reversed Order 


803 
783 


348,. , 

V 

312 


43.3 
39.8 


62 
47 


12.0 ^ 
9.1 


1 36.2 

1 

|32.0 


Expectation 
Letters 


Letter, Trainee 
& Supervisor 


756 


335 ' 


44.2 

■ 




11.9 


33.7 


* 

t 
I 

< 


'No Letter, Trainee 
" & Supervisor 


>. 

828 


325 


39.3. 


' 52' 


9:4 


32 1 7. 


I 


Letter i Trainee 


471 


* 

160 


33.9 




5.8. 


-30 :o 




- » 

No Letter, Trainee 


525' 


150 


28.6 


24 j 


i 

6.0 i 


34.4" 


• 

1 ' 


Letter, Supervisor 


287 


175 . 


61.0 


38 1 


■"1 
25.3 j 


37.0 


V ' 1 

■ A -J. 


No Letter, 
Supervisor 


! 

303 1175 

i 

•! i 


1 

57.8 j 

1 


1 
1 

28 1 

I 


i 

17.9 

* 

1 

i 


31.3 



! L 
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. TABLE 6. CORRELATIONS BETWEEN 
RATING &CAL! 




i6E RATIHGS FOR "DIFFERENT 
SHORT QUESTIONNAIRES 



"Instrument 



Frequency With frequency With Criticality With! 
Crit4cality , 'Performance Performance i 



I Short - 

i Questionnaire 

.1 V 



.93 ... 



.97 



.94 



( 












: Long, 






• ■ :> 






j Questionnaire 

1 




.70 


.84* ■ 


.70 


♦ 

A 

1 



The patterns of average ratings for itejiis- on tfie- Perfonfifince of 
Task Upon Arrival From. School sca>e for the long questionnaire and card 
sort were.nearly-ldentical, the correlation between the two instruments 
for the 134"items being .89. The Qiagnitude of this correlation implies 
tnat the differences between average ratings on performance of different 
items are r'eal differences and not chance occurrences. ^ - ^ 

u j^*'^°'""^"ce ratings for several of th'e items lhat were not present in 
the drid Radioman "A" School curriculum were significantly ^ower than the 
average Performance rating for all items. For exampla, item 4 on the 
long questionnaire, which dealt with reading perforat-efd teletype tapes 
_ha_d_an average rating of 2.15 which was significantly- lower than the 3.31 

• overall average rating of Performance ( t = 7.9, df'= 390, p<.001). In 
addition, the nine items that dealt with-the use of International Morse 
Cod^ (itelins 122 through 130) produced an average performance fating of 

^2.45, wMch Wcfs significantly lower than the overall PeffAjemance average . 
(t = 7.7,,df = 390, p<.0Ol).. ■% ' ,^ ^ , 

SHIP/SHORE DUTY STATION DIFFERENCES. The second largest source of variance 
^ in the rating scale data was»'TDroduced by the duty station of the'trainee 
^ (see.tabTt 7J. Oij both the^requency of Task and' Critica-lity of Task 
scales significantly higher ratings were found fjor shif personnel than ' 
shore perspnnel .(t = IIJD, df = 658, p^.OOl for Frequency; t = 8.6, 
df = 658,>p^.00T for Criticality). However, average ratings. on the - • ^ 
Perf^mance'of Task Upon ArrivaT From School scale were nearly Identical - 
for shore biased and shipboard ,persoBn^l . ^ 
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TABLE .7. AVERAGE SCALE RATINGS OF SHIP AND SHORE DUTY 

STATION RESPONDENTS, TRAINEES, AND SUPERVISORS' 



Respondent"' 

y- 


* 

Frequency of 
lasK bcaie 


. Critical ity of 
lasK Scale 


• Performance of 
Task up6n Arrival 
from School Scale ^ 


Shore Duty^ 
Station 


- 2.64 


•3.33* 


• - V 

3.18, • ■ 


Ship Duty .> : 
Station 


V 

, ^ • . 

'3.31 


3.88 


3.19 ■ . 


Trainee 


3.02 


3.62 


-> 3^34 , 


Supervisor 


2.95 


' 3. 58 


3.05 



Correlations between shipboard and shor^-based p«sonnel for the 
-^average item ratings on the different scales. of the long questfonhaire 
. were .72 for Frequency, .70 for Critical ity, and .88 for Performance X)f 
Jask*llpon Arrival From School. The Performance correlation was significantly 
higher (ji'<.05) than the F^^equency and Criticality correlations. The 
high Performance correlation reflects a high degree of" similarity of ^ 
Performance ratings for the two groups, which would be'|expected since 
both received the same training.* 

. The large ship/shore diffe^rences in ratings of Frequency for many 
items correspond to the known differenqe? in the job st^r^dcture for 
Radiomen between these different sites. The accuracy^pf these Frequency 
ratings was used to evaluate different groups of respondents on their 
instrument completion performance and these results are, discussed later 
in this section. 

TRAINEE-SUPERVISOR DIFFERENCES. As shown in table 7, thS^bWall average^ . 
ratings on the Frequency and Criticality scales were nearly ^identical 
for trainees and supervisors. However> trainees provided significantly 
higher average ratings on the Performance^ of Task Upon Arrival From 
School scale than supervisors (t = 4.9, df = 658, \><^.wt)J Despite 
this difference, the basic pattern /of item ratings was highly similar 
for the two groups. This similarity was shown in th^ correl ati/ons 
between; trainees and supervisors on the average Performance ratings^ for 
items which were greater than .80 for both fhe long form and card sort / 
and greater than .90 -for the' short questionnaire.,/' s , 
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Some tasks included on the questionnaire are performed very often 
by Radiomen and some are seldom performed. These tasks were a source of 
rating differences between the trainees and supervisors, with supervisors 
tending to give the more extreme Frequency ratings on such tasks. Such 
items are included in the Frequency-Accuracy Index discussed below. The 
more extreme Frequency ratings for these particular items corresponded 
to the known frequencies -for these tasks, and supervisors were more 
accurate than trainees in describing these known frequencies. ' 

CLASS STANDING. Class standing ratings are shown in. table 8./ Statisti-- 
cally significant differences in average Performance of Task Upon Arrival 
From School appeared among the different groups. Graduates in the upper 
third of ^he classes and their supervisors rated Performance hi-gher than 
graduates^and supervisors of the other two groups. The difference 
between^ the upper third and middle third was significant (t = 3.4, df = 
462, p<C.001), as was the difference between the upper third and lower 
third (t = 3.6,j^= 436, p<.001). The difference between the middle 
third and lower^^r«j was not significant. 

TABLE 8. AVERAGE SCALE RATINGS FOR CLASS STANDIf/G GROUPS 



Class Standing 

> 


Frfiquency of 
Task' Scale 


Critical ity o/ 
Task Scale 


Performance of 
Task Upon Arrival 
from School ScaT&\ 


Upper third 


2.99 


3.65 


3.35 


Middle third 


2.94 


3.57 


3.12 


, Bottom third 


^ 3.02 


• 

3.,.58 


3.09 



TIME LAPSE FROM GRADUATION. There Were significant differences in* 
Frequency of Task ratings dependent upon the ampunt of time lapse from 
graduation -prior to interrogation (table ^). The 4 to 6 month group 
provided significantly lower average Task Frequency ratings than the 7 to 
month group (t = 3.0,- df = 286, p<.OT). In turn, the 7 to 9 month . 
group provided significantly lower average Task Frequency ratings 
the greater than 12 month group (t = 1 .7, df = 411 , p<.05). The/rend 
is clearly one of higher ratings of Frequency of Task with more time lapse 
from graduati'on. Differences between , these groups on the other scales 
were not -significant. 
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TABLE 9. AVERAGE SCALE RATINGS FOR TJME FROM GRADUATION GROUPS 



. Time lapse 
Prior to 

Evaluation 
\ 


Frequency of , 
Task Scale 


Cri-ticality of 
. /Task Scale 


Performance of 
Task' Upon Arriyal 
From School Scale 


4 to 6 months 


2.71 


• 3.51- ^ 


3.27 


7 to 9 months 


2.99 


3.61' *" 


3.oa 


10 to 12-months 


3.02 


3.66 


■ .3.18" 


More than 12 • 
, months 


3.12 


'3.66. 


3.19 



■INSTRUMENTS. Rating scale data for the different instruments are presented 
in table 10. The short questionnaire produced significantly- higher 
average Frequency of Task ratings than the long questionnaire (t = 5.4, 
df = 461, p^.OOT). On the Performance of Task Upon Arrival From. School 
scale the average rating .from the short questionnaire was significantly 
lower than the loag questionnaire (t/ 5.6, df = 461, p<.001). Both of 
these diffejiences probably reflectecrthe different items that jnade up ' . 
the two differerrtHHt^stcyments and were not related to the length or any 
other general characteristt6s,^f these instruments. The absence of a 
difference in Performance rating? for the long form and card sort would 
be expected since both utilized the same 134 items. 

TABLE 10. AVERAGE SCALE RATINGS FOR DIFFERENT INSTRUMENTS 



Instrument 



Short Form 
Long Form 
Card Sort 



Frequenc^f^ of 
Task Scale 



Critical ity of 
Task Scale 



3.17 
2.78 . 
N/A 



Performance of 
Task Upon 
Arrival From 
School Sqale 



3.68 • 



N/A. 
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ITEM ORDER.* Average Performance -on the normal item order instruments 
was lowen than average Performance on the reversed item order instruments. 

.However, this difference was caused primarily by the significant difference 
between the average Performance ratings on the nomlal order (2.79) and 
reversed order O.09) short questionnaires (t - 2.9, df 269, p^. 01). ^ 
The correlation between the normal and^'Teversed forms for Performance 
ratings of the 15 short questionnaire' items - was .94, which indicates 
that despite the difference in level , /relationships between items were 
nearly. identical for the two forms. The normal item order and reversed* ' 
\tem order ■:yers ions of the long questionnaire also produced nearly 
identical data. The correlation between the two versions was .89 for 
Perfo'rmance' ratings. o ^ * 

*» " • "■"'''^ 

TABLE^'ir: AVERAGE SCAL& RATINGS FOR ITEM ORDER, ADMINISTRAflVE 
, PROCEDURE, ANKeXPECTATION LEHER CONDITIONS 





Instrument I 


Frequency of 
Task Scale 


Xnticality of • 
* Task Scale 

— ^ 


Performance of 
Task Upon 
Arrival From 
School Scale 




Normal Item ^ 
Order ^ .,y 


^ 2.95 ■ 


3.62 


3.12 




fever$etf*Item 
Order 


t 

^- 3.01 


3.58 


3.27 


X „ 


Single Admin. ^ 


f - . 
2.99 


'3.63- , 


3.21 




Joint Admin. 




, 3.57 


.3.16 




Expectation 
Letter 


I- ' ' ■ 
' 3.00. 


3.59 


3.26 ' 


A 


4 

\ No Expectation 
Letter 


'2.97 


i 

3.6T 


3.15 



> ACCbRA(^Y OF FREQUENCY RATINGS. As not^ earlier," the overall average 
«X-frequency rating was higher for shipboard personnel than for personnel 
at shore stations. This difference was. produced in large part by eight 
items related to operation of the Fleet Broadcast and by four it^ms 
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related to maintenance performance on conmunications equipment. Other 
items such as those related to radiotelegraph operation had been revealed 
by task analysis to have very low frequencies for both ship and shore ' 

^personnel. Two items related to security practices were also known to 
have very high frequencies for bptIT ship and shore personnel as did a 
number of items have low frequencies of occurrence only for shore-based 
personnel. From all these tasks with previously ascertained very high 

•or very low. frequencies of occurrence for ship personnel, shore personnel, 

- or both, 7& were selected to provide a measure of the relative accuracy 
of Frequency ratings .for different groups and conditions of the study. 
On each item comparisons were made for pairs of conditions, such as 
'supervisors vs. trainees. If the average Frequency rating was more 

extreme (and therefore more accurate) for the trainee, he received, a one 
for the ijbem and the supervisor, received a zero. This was done for all 
78 items and a total score for each condition was obtained, with the 
Gondii on having the higher number being more accurate. 

When average ratings for trainees on these 78 tasks were compared 
to the average ratings for supervisors, only 21 of the 78 items were 
rated more accurately by the trainees. The sign test (Siegel, 1956) 
indicated that significantly more items were rated more accurately by 
supervisors. (The correlation between trainees and suoervi'sors for the 
134 Frequency ratings was .90 for shipboar^ personnel and .93 for personnel 
■ at shore stations. Thus, despite .reduced accuracy of responding, trainees 
still nearly replicated the same inter-item relationships produced by 
supervisor ratings. ) 

The joint vs. single administration factor produced a significant 
difference in the accuracy of Frequency ratings. Fifty-one of the 78 
tasks were rated more accurately by persons receiving the long question- 
naire jointly with the supervisor (or trainee) than by persons in the 
• single administration condition (p<.01, sign test). 

J? - 

Another significant-difference in Frequency rating acclira.cy was ■ 
found for the lower of the three Class Standing groups. »' Average ratings 
for this group were significantly less accurate than the overalls-ratings 
on 53 of these 78 items (p<.01, sign test)* The middle third and upper 
third groups did not differ between themselves,' but average ratings of 
bath were more accurate than overall average ratings on 50 of the 78' 
items (p.<.01). 

Two of the Time from Graduation groups produced significant differences 
in 'accuracy of ratings for these 78 items. The group who received 
instruments 4 to 6 months after graduation was significantly more accurate 
. than the overall average on 55 of these items {e<.01). The. group that 
received instruments" 10 to 12 months after graduation was le*ss' accurate 
than the overall average on 53 of, these items (^.01). No other 

- - • '41 , . 
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differences between groups and conditions in the Study appeared for this 
measure of the accuracy of Frequency ratings. 

STRUCTURED INTERVIEW SIMILARITIES AND DIFFERENCES. The patterns of i.tem 
ratings were nearly identical between the interview and mail -out data, 
expecialTy in the case of ship personnel. The correlation between 
Frequency ratings from the, two procedures was .74 for shore-based person- 
nel and .94 for ship personnel. For Criticality ratings, the correlation 
was .68 for shore personnel and .89 for shipboard personnel. For Perform- 
ance upon Arrival from School, the correlations were .89 for shore 
personnel and .92 for shipboard personnel. The lower Frequency and 
Criticality correlations for shore personnel reflect the somewhat atypical 
equipment and procedures encountered it the Norfolk Communication Station. 
As with rating averages, variability of ratings was also nearly identical 
for persons receiving mail-out instructions and for persons providing jg^ 
data in the structured interviews. - ' ^ 

One difference which occurred between ratings for structured inter- 
views and mai.l-out questionnaires was a greater tendency/or the structured 
interview group to skip the Criticality and Performance seal es if the 
task was "Never Performed." It was difficult to insist that a person 
make a Performance rating on a task when he (or the person he supervised) 
had never performed this task on the job. Mail -out questionnaires 
contained instructions to do this and respondents' were ggneruHy willing 
to provide such ratings. In the face-to-face interview, the problem of 
making these ratings was a shared problem, and the joint decision was 
often to not make such ratings when there was little or no experience 
upon which to base them. 

TRAINING ADEQUACY INDEX. Tasks with high ratings of Criticality and low 
ratings of Performance Upon Arrival From School are tasks which are apt 
to be undertrained. Conversely, tasks are apt to be overtrained if they 
receive low ratings of Criticality and high ratings of Performance. The 
134 tasks' of the long questionnaire were examined and those with the 
largest differences between average Criticality and Performance ratings 
were identified. Tasks with large positive differences, i.e., greater 
Criticality than Performance, were classified as' undertrained on this 
Training Adequacy Index. Tasks with large negative differences; i.e., 
greater Performance than Critical'ity, were classified as overtrained on 
this index. This procedure corresponds closely to a technique developed 
by Siegel, Schultz and Federman (1961) for combining Task Criticality . 
and Task Performance ratings to obtain a measure of training adequacy. 

Since training problems in the old Radioman "fi^" School curriculum 
centered around the duty-station factor, the proce^dure was 'carried out 
separately for average ratings provided. by trainees and supervisors at 
shore duty stations and for average ratings provided by shipboard personnel. 
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This pirocedure indicated four tasks (items 14, 36, 48 and 72 in appendix 
B) which personnel from both duty stations classified as overtrained, 
and six tasks (items. 74, 75, 102, 103, 123, and 124) which personnel 
from both duty stations classifie(j[ as undertrained. Shipboard personnel 
classified an additional eight tasks as overtrained (it^ms 15, 35, 50, 
51, 63, 67, 108 and 115) and an additional eight tasks aS. undertrained 
(items 3, ?0, 94, 101, 110, 122, 127 and 128). Shore-based personnel 
classified an additional nine tasks as overtrained (items 40, 41, 43, 
45 , 46 , 47,. 82, 83 and 84) and an additional seven talks as undertrained 
(items 4, -57, 112, 131, 132, 133 and 134). Agreement of the two duty 
station groups on only 10 of the 42 items augurs well for the derived 
Training Adequacy Index, considering the large differences in training 
needs, for the two duty station •groups and the fact that the old curriculum 
was identical for' both. . 

The tasks rated als overtrained by both duty station groups involved 
those dealing with assignment and recording of date-time groups of 
messages, delivering messages, and changing teletype paper, tapes and 
ribbons. Unlike many training tasks which receive little attention in 
school, these tasks are practiced frequently since they are included in 
performance of many other training tag^s dealing with message handling 
and teletype operation. For this re^n, these tasks are overl earned; 
i.e., more practice is given on the task after criterion oerformance has 
been reached. However, this overleaming is a by-product of other 
essential training. 

Of the items which both duty station groups classified as under- 
trained, one dealt with rescuing a person from a live circuit, one with 
first aid, two with distress messages and enemy contact reports, and two 
with restoring fading communication links. All can be considered emergency 
situations and at first glance appear to be likely candidates for increased 
training emphasis. However, no particular increased emphasis is placed 
on these tasks in the new Radioman "A" School curriculum over that of 
the old curViculum. 

For personnel at sh'ore. duty stations the derived Training Adequacy 
Index indicated overtraining for tasks involving identificatibp* of call 
signs, operation of the Fleet Broadcast (five items), operation of 'two 
teletype terminals and operation of a receiver. For the most part th6se 
are procedures and^equipment that are used primarily aboard ship. Shore- 
based personnel now' do receive much less training emphasis on these 
items under the new tracking system in Radioman "A". School. 

♦ 

Personnel at shore duty stations classified tasks as undertrained 
which dealt w^ith reading .of perforated teletype tape, operation of the ^ 
Autodin terminal, conduct of emergency destruction of classified material 
and five other items related to security practices, Reading perforated 
tape and AutOdin are items in the new Radioman J'A" School curriculum 
that were^not included in the training of the persons in our sample, 
nfe tasks related to security practices are especially critical for . 
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shore installations which usually are in foreign countries, Undertrainirig 
is probably the case for these tasks, but since the procedures are 
generally specific 'to the particular Units, more on-the-job tr-aining, 
not more school training, is indicated. 

Items indicated to be overtrained for shipboard personnel included 
the logging of outgoing messages; identification of duplicate messages, 
special messages and readdres'^d^mes sages; distinguishing between ship 
and shore teletype circuits; and distinguishing among the three major 
components of . a shore conmuni cation station. Also included were, tasks 
related to placing tapes in backlog bins, operating a particular reper- 
forator, and pointing antennas. All of these "overtrained" tasks except 
painting antennas are primarily shore-station functions which now do 
receive much less emphasis in Radioman "A" School for ship-bound trainees. 
Antenna painting did not receive much attention in "A" School in the 
past and still does not. The overt^^aining indicated by the Training 

Adequacy Index may have been due to the simplicity of the task- 

* * * 

Undertraining was indicated by the index for shipboard personnel 
for tasks of checking the accuracy of prepared tape^ identification of 
incorrect Naval Activity Short Titles, activating crypto equipment, and 
use of ddUWter measures, and otTier procedures related to enen^y jamming. 
Short Title identification is a new feature of Radioman "A" School that 
was not included in the training of our sample. More emphasis is also 
now given to operation of crypto equipment for shipboard personnel. 
However, jamming procedures receive less emphasis in Radioman "A" School 
now than in the "past and undertraining indicated by the index may be a 
"false alarm." 

In summary, the derived index provides generally good data on the 
training problems known to exist in the old Radioman "A" School. Where 
it fails, it is usually because the overtraining indicated is unavoidable 
and not a school problem, or because the index does' not distinguish 
between undertraining that is a school problem and undertraining that 
must be corrected by on-the-job training. The moderate success of this 
derived Training Adequacy Index speaks well of the reliability and 
accuracy of the Critical ity and Performance ratings from which the index 
was derived. 

OPEN-£NDED QUESTIONS 

For all instruments compared in the study, open-ended questions 
were used to identify mi^ssing objectives in the. Radioman "A" School 
curriculum. Six different categories of missing objectives were mentioned; 
(1) use of publications, i2) perforated tape reading, (3) radiotelegraph , 
procedures (International Morse Code), (4) NAVC(^PARS (computer communica- 
tic^n procedures), (5) satellite communications, and (6) voice communications 
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The percentages of ^p^r sons mentioning these Items for the. different 
instruments and_for different respondent groups ar^ presented in table 
12. Three of the si)f^ categories of missing objectives have been incorpor- 
ated in the new Radjoman "A" School cfirriculum. and tentative 'plans are 
being made to include a fourth (voice communications). -Discussions with 
school personnel -indicated that use of pubTications may also be a bona- 
fide raissing objective although, until we reported our results, it was 
not considered for inclusion in the curriculum.. One scale ratlog which, 
appears to be a "false alarm" by the questionnaire instrument may, in 
fact, not be. Radiotelegraph procedures are taught in the more advajice^ 
communication schools and presumably are related to the jobs of,' higher 
rated p_ersonnel ,_Sinci these procedures were targeted as missing trainlTig, 
It indicates that some "A" School graduates are being placed in'work ' 
situations requiring that skill. 

A high percentage of persons receiving structured interviews indicated 
the task of reading perforated teletype, tape as a missing training 
objective. These responses were mainly from personnel at Naval Communica- 
tion Station Puerto Rico where this task is frequently performed. This 
item was largely responsible for the greater average number of missing 
objective comments for the structured interview group. However, none of 
the differences among instruments was significant nor was the difference 
between supervisors and trainees. 

The short questionnaire depended on open-ended questions to identify 
specific areas of undertraiping and overtraining. Twenty different 
tasks received mention as b^ing undertrained. The most frequently 
mentioned item was crypto equipment with 15 percent of respondents on 
the short questionnaire reporting it as undertrained. The least fre- 
quently mentioned item^as first aid which was mentioned by .only one 
percent of the shprt form respondents. Tasks mentioned by .tjve percent 
or more of respondents typically were indicated by other soWces to be 
tasks that were undertrained. These sources included the changes in the 
old curriculum for the ta^k and also data from the derived Training 
Adequacy Index discussed p'reviously. When 'an item 'was mentioned by 
fewer than five percent of respondents, the other sources usually indi- 
cated adequate training or. even overtraining for the item. First atd 
was one of the areas that the derived Training- Adequacy Index ind-icated 
to be undertrained*, yet only one percent of respondents mentioned this 
item: It is not clear whether this reflects on .the open-ended question 
as a source of information on undertraining, en the derived Training 
Adequacy Index, or on both. ' " « - 

SigiMficantly, fewer comments were provided about overtraiirrng than 
were made"^'about undertraining. The only area mentioned as overtrained 
by more than three respondents was shipboard equipment and systems. 
Sinqe practically all of these overtraining comments came from shore 
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• TABLE 12. 'percentage OF RESPONDENTS INDICATING MISSING' 

• OBdECTIVES ON OPEN-ENDED qyfSTIONS FGR^ DIFFERENT' 
INSTRUMENTS AND GROUPS - ^ 



Reported 
Missrng i 
Ob^ectivds 




A. 



<s ■ 



il 



1 ^ 


Nt96 


N=196 


N=197 


N=267' 


N=387 


. 1 
N=369 1 


Publications 


5 


9 


12 


iM 


19 


, 5 


Tape Reading 


29 


.10 


7 


14 


•'.8' 




Radiotelegraph 


7 • , 


• 4 


2 


6 


6 . 


1 , 


NAVCOMPARS 


4 


2 


2 ' 


3 


3 


2 


|SATCOM ' ^ ' 


1 


2 


2 


'. 1 


2 


0 


jVoice Communication * j 


2 . 


; .2 


2 


. " 6 


4 


3 


'Average Percent per \ 
Item 

: ' • 1 


'8-.Q 

i 


. 4.8 


4.5 


i 

7.0 1 6.8 

\ r- 
1 

• 1 


1 



^4 
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installations, these comments appear to be valid. Again, only. a small 
percentage (eight) mentioned this item, but it is indicative of the 
sensitivity of the instrument, 

FOLLOW/UP LETTERS JO RESPONDENTS AND NON-RESPONDENTS 

To obtain opinions from Fleet personnel regarding this particular 
feedback effort and about the process of obtaining feedback information 
by the schools, follow-up letters were mailed to the 1026 trainees and 
supervisors who had not returned the questionnaire and to the total 
sampre of 600 trainees and supervisors wfio had responded. Only 145 
letters to non-respondents were returned." An analysis of the reasons' - 
given for not returnirrg the feedback materials indicated that the RadiOT 
men considered the materials too long^(15 percent of returned letters, 
mostly from trainees), did not receive the materials (18 percent),- Tost 
the materials (10 percent), did not feel qualified -to evaluate "A" 
School (11 percent), did not understand the questionnaire (5 percent),* 
were transferred (6 percent) , and did not use Radioman "A" -School training 
(9 percent^. It is interesting to note that 59 percent of those reporting 
that they' either did not receive -or lost the materials were mailed the 
long questionnaire. This is significantly different than the 33 percent 
that would be expected by chance (z = 6.1, p^l.OOl). 

! Of the 600 follow-up letters mailed ta^ respondents, 175 were returned 
A number of trainees receiving the materials within the time frame o.f 10 
or more months following graduation from school indicated difficulty in 
remembering the school curricula and generally concurred that the evalua- 
tion should be conducted sooner after graduation (23 percent). In 
addition, 27 percent of the respondents felt that the evaluation forms 
should be tailored to the tasks requifecl at ship and shore duty stations. 
Somfe supervisors (5 percent) expressed doubt as to their ability to 
evaluate the trainee based on the "A" School curriculum or to remember 
trainee performance upon arrival at the job site. 

The remaining trainees and supervisors indicated no problems in 
completing the materials. A number of personnel indicated their pleasure 
at being asked for their opinions aiid complimented the format (11 percent) 
Some respondents indicated a desire to receive feedback on the results 
of this study and future evaluations. '^^ 

A small number of respondents indicated that they did not appreciate 
the critical ity of the retuf^n of evaluation materials until they received 
the^ipDllow-up inquiry. .Such comments support the importance of an 
expectation letter, in order to insure Jhat the respondent understands 
the value of his input to the training cycle. - / ' • 
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SUM1ARY OF RESULTS. . ^ , . • • / ' ; 

RETURN RATE DATA. \ ' 1: . /. ^ r 

. ^ Return rate for supervisors was aTipos-t doiible that, for trainees^ 
(59.3 percent vs. 31.1 percent). . / 

Return rate following* reminder letters, was greater for super-" 
visors than trainees (21.6 percent vs. 5.9 percent). • 

. , Return rate for th& short questionnaire was 'greater tlian for.., / 
the long questionnaire or card sort.^(48.9 percent,; 37. SLperceaft, 
, ' ' and 38.2 percent,' respectively). S --^ . - / ' 

. > ' , V. : ^- \ ^ 

Time until return of questionnaj^Sg^iwas^ longer for the long ^. 
* form than for the short form €iml' card sort (38\8*daysv 30\7 
days, and 33»2 days, respectively)./ * - • ^ * . V.-' 

For trainees, the return rate for the short fonn was greater 

than for the long^form or card sort (40.0 (ier cent, 25.3 percent, 

and 27.7 percent, respectively). . 

« 

The -group 4 to 64nonths from graduation yieldedothe. lowest 
return rate for the time from graduation groups (4 to 6,^ 35.3 
percent; 7 to 9, 41.2 percent; 10 to 12, 41^.5 percent; greater * 
than 12, 43.8 percent), ' . 

Trainees graduating in the Upper third and Middle third of 
their classes had a higher return rate' than the Lower third 
(35.1 percent, 33.3 percent, and 23r9 percent, respectively). 

/ 

The group receiving '^Expectatiop lietters" had a ♦higher return ^ 
rate thaji thoSe who did not. receive them (44.4 percent vs. 
' 39.0 percent). • . • . , 



RATING SCALE DATA. 



Correlations between rarting« scales were higher, -for short. ^ 
forms: 

Short Form Long Form . Variable 

'.93 .70 Frequency ^ith Criticality 

.97 .84 Frequency with Performance 

.94 .70 Criticality with Performance 
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• * 

Higher ratings were given for shipboard personnel than for 
shore-based personnel on both Frequency and Criticality of 
Task,scales (due largely to the inclusion of more tasks specific 
to shipboard duties on the instruments). 

. Average ratings for the Performance ,<)f Task Upon ^Arrival Front 

School strile were neaFl_y Identical for shipboard and shore- 
• based personnel'. 

Correlations between shipboard and ^hore-based~persorinel for 
' the Frequency, Criticality and Performance scales were JZ, 
.70, -and .88, respectively. 

.Overall average ratings for trainees and supervisors were 
nearly identical on the Frecftieiicy Tind Criticality scales. ' 

Trainees rated^' their own Performance higher than supervisors 
rated this performance, but the. basic pa'ttprn of ratings 
across tasks was similar. Be correlations between the Perform- 
, a.nce ratings of the two groups was greater than .80 for both 
.the long form and card sort and greater thair .90 for the short 
questionnaire. 

, \ 

Trainees graduating in the 'Upper third of their classes -and 
their supervisors r*ated trainee Performance higher than graduate* 
, land thel^ .supervisor^) from the Middle and Lower groups. 

The correlation for; average Performance ratings- on the short 
form between Rornial and reversed item formats was ' 

The correlation for average Performance ratings on the long 
form betwTeen. normal ajid reversed item formats*was, .89 

Supervisor ratings of Frequency were more accurate than those 
of trainees. - . ' , 

Average Frequency ratings were more accurate under the joint 
administration condition thafi under the single' administration 
condition. * • 

Trainees in the lower third of their graduating classes were 
less-accurate in their Frequency, ratings than trainees in the 
Upper and Middle thirds. ^ 

^'•-x^ ■ - .• 

The group 4 to 6 months from graduation gave more accurate 
Frequency rati ngs t han* other groups * ^ , 

The groiJfj^'lO to 12 TOhths^from graduation gave less accurate 
Frequency^ ratlngs^han , other groups. 
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A Tra1n|ng Adequacy Indent based on Criticallty of Task' and- - • 
" Performance of Task Upon Arrival From School,, accurately 
identified many tasks, which were previously known to be pyer- 
trained or undertralned. " ■ . • ' y^- 

OTHER FINDINGS. ' ' ' ^ . * ; T ' . , 

Open-ended questions identified training objectives that were^ 
missing from the old curriculum that have been included in the- 
newf qiirriculum. 

Follow-up inquiries elfcited the following major information: 

^' " ' ' . ' ' 
59 percent qf those reporting loss or' non-receipt of question- 
naires were mailed the long form> Only 33 percent would be 
* , expected by charrce and the difference was highly significant. 

Of tho$e who expressed difficulty completing the short form, . 
23 percent expressed an inability to remeufcer.the school 
curriculum as a reference point. ■ ^ ' ^ ' " 

A number of respondents e?cpressed a desire that future feedbacl^^ 
questibnnaires be speciinic to either ship or shore duty > 
requirements (27 percent). 



X4' 



r. 

\ -50 



TAE6 Report No. 19 



' ' SECTION IV^ 

DISCUSSION AND CONCLUSfjO^NS < ' 



moi 



- , The most comp^^lling kpect of the results of this studV was the 
consistent patterns of ave?s;ge item ratings that ware found onihe 

ta? %on Z^^'i't l'^b^'^'V■'^V'' ''''^ '"^ PeVforSa"nce''of • 
rdb^ upon Arrival From School. SimiTar patterns Were DroducpH hv oarh 

?fiTErSL:rP°"^'?*' ^^^^ adniinistratJon prbcelre o?w 

te?w^e^ aJm;!???J"^-°'''^'*^"5"^ calculated between 'different gZlslk 

^Jf Th?s h aS'rp;°j;Hp'^K-?-r" f ^ many wer/fgO dr 

ifo? OrouDs whJref"'"';^'*^°^ '^"''"9^ ^te'"^ ^3ti"gs Qccurred.even 
|Pr groups m which the number of respondents W^s 50 or less. 

highly°Jel?lbir'L^'thfrr^' ''^'^^l ^^^'^^ °" .ther different scales 
. Miyniy renaDie, but they corresponded to the diffprpnV-Pc %hr»f ti«v,« 

' c'???ica?i?J''1nS^l°^-'° adminiWati-oT?n1re^^ 
critica ity, and trainee performance for these tasks- The larae^hin 

'11^^^ frequency, which were the ksl fo? the fe^^nt 
■•S?prJo7-''"Irf^'^''^*'°" °^ '^^dioman "A" School, are acJuf'ately s 

^ "Ne'Jef rfSrSd-teTr ''V.^'t ''J^"^'''^ ^asks'were ihdicated as ^ 
wever perTormed py^early all shoVe-based pet'sonnel and «;imiiariu 

real-wo?l?s"!'ti;a?ioT-"?hI Ij^f.Jhe Frequency ratings', reflected the ' 
re0i world situation. The ability to r'read perforated teletvoe tanp.; 
important for many Shore-based personnel, but-' this task wis not v"^ 
included in the curricu>um of those persons interrogate-^ in our {?.!hJ ^ 

e?yT:"or°tys'^\sT" ^ '-T/?'^' rali^gfle'e^oSnS L't' . " 
very low Tor.tnis task. Items reiated to International Morse TndP ik^p 



51 



. * * / '.TAEG Report 19 

: ' ■ • ' ' ' ■ • / ■ 

•performance typically increases as trainees gain more experience an'd 
Frequency ratings showed significant "increases for groups with longer 
time from graduation. This finding also indicates the validity of these 
average ratings. * ^ . 

The high r§];iabil.ity and high validity of the present rating scale* 
data indicate ;bfiat mail-out question.nairses pan provide Excellent data 
that Can serve'^as a means for identification of training, problems. This 
finding is contrary to an expectation that* developed eanly io >the study 
that only -expensive face-to-face interviews • with personnel of operational 
junits could provide effecJS^ive xiata for these purposes. This bias was 
generated by a number of published and unpublished studidj which reported 
unsuccessful results using mail-out questionnaires -(e.g. , TAEG Report . 
No» 12-1, 1973). With minor modifications^ questionnaire format and * 
administration procedures used in tills study appear to be applicable to 
nwst, If not all, Navy training situations. The questiorinaire^nd 
procedures are described in Section V of thi.s report and are described 
in more detail in a procedure manual which is being published separately 
as TAEG Feedback ManuaL (June 1975). - ^ 

The structured interview technique, which elicited our initial 
faith, was also examined ^n the present study but, as reported earlier, 
no basic, differences bety/een the structured interview and Tong question- 
naire were found in the ratings of job tasks *qr in the responses to the 
open-ended questions. In fact, the highest correlation found in the 
study was between Performance ^ratings on the long questionnaire and 
Performance ratings on the structured interview (.94). Variability of 
item ratings also did not differ between ratings obtained from structured 
interviews and those Obtained in the mail -out questionnaire jlong form). 
Tjje similar variability of item ratings indicates, that ojie returned 
questionnaire vis equivalent to one structured interview. Although the 
structured interviews enabled^the gathering of inform'ation that was not 
specifically requested in the questionnaire, this information could have 

, been obtained by. mailing a second questionnaire. . The struqtured inter- 
view should be used during the development of 'the mail-out instrument. 
Interviews could be conducted with training personnel, especially those 
personnel recently assigned from operational units. This will allow 
identification-of questionnaire amblguitiesi and identify additional , 
useful information that should be requested in the questionnaire (see 

-sectien V) . \ . ' • . 

As was noted earlier, statistiqally significant -difference's were 
foOhd in the accuracy of ratings for* different respondent groups and for 
different procedural conditions in the present study. However, despite 
Jthese differences, very high correlations existed between ratings of 
"moFS accurate and less accurate groups.. For tills reason, only the 
lar'gest difference In rating accuracy, which appeared between supervisors 

\. ' ^ 
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and trainees,. would influence these procedures for obtaining -post 
formal training feedback. Trainee/supervisor differences in ratings for 
an itejn often occurred due to ambiguous task descriptions or because a 
piece of e^qufpment was identified^ by nx)del number (e.g., AN/SRC.-20) or 
other. name that conveyed little infomjation about it or its use.- .Under 
these circumstances, supervisors w^re'moretapt than trainees to give 
Frequency ratings that were consonant with the known frequency of perform- 
aYice of the task. This result implies that more weight should be given 
to supervisor ratings on- task items when these ratings differ from those 
of the trainee/ The .appearance of trainee/supervisor 4ifferenc!.es for 
ratings on a task may also indicate that the item requires revision *- 
prior to the ngxt series of mail -out questionnaires. ' 

The large differences in return rate among the different types of 
mail-out instruments were directly .^r-elated to the length of the ijistru- " 
meht. Follow-up letters -sent to persons who did not. return tfie long^ 
questionnaire often were returned with the "Questionnaire was too long" 
category checked. The^advantage of the short questionnaire was particu- 
larly striking for trainees. More than one. and one-half times as many 
short forms- than lojig forins were returned by this group. * Fx)r supervisors, 
only 10 percent more short' forms than long forms were, returned. These ' 
return rate differences between long. and short instruments were closely- 
paralleled by differences in time" to return these instruments, with more 
time required for return of the long form.' , • r 

Although the-return rate was higher for the , short questionnaire, 
much less information about training problems resulted from this instrument 
Often, the general objectives listed in the short form were rated as 
haying "Adequate" , performance whereas on the. long form some enabling 
objectives subsumed under the genera] objective were rated "Substantially 
Inadequate." To a limited extent, the lack of specificity of the short 
questionnaire was compensated for by open-ended questions which were 
included-to uncover specific problems. However, such a 'small' fraction 
of respondents mentipned any particular problem that it is difficult to 
assess the extent , of the training' problem. With the long questionnaire, 
the specific item is rated by all respondents and the training problem 
appears prominently against the background of non-problem items. 

Considering the difficulties in identifying specific problems with 
the short questionnaire, it appears that the factor of nine that was the 
ratio of items between the long and short forms (134/15) also described 
the ratio of . information provided about training problems. Other evidence 
for the poor' information source of the short questionnaire was tlie near 
perfect correlation between scales on the short forms (see table 6). 
This indicated that no more information was communicated by ra\:ings on 
all three scales than was comnunicated' by ratings on any one of the 
scale's. . ' 



^ differences appearsNj between the normal -itfem-order long question- 

^£:^iit4^ the reversed- item-order long questionnaire. In fact, the' 
^j^rf0ption of .90 between Performance ratings on these two forms was 
^M0!^Wt\\^ highest ccfrrelations that appeared in the study for the 
P]^,|ft>^ance scale of'^the long questionnaire. This Indicates theft question 



^ igjX|.~accuracy was constant or nearly constant throughout all 134 i-tems, 
/^ri(t\£^at even longer questionnaires can be Used, if necessary, to provide 

'f ^._^St^ was noted in the previous section, the differences in return 
\ rate between trainees and supervisors were large with nearly twice as 
, many questionnaires 'returned by supervisors as by trainees. The. higher 
return rate of the supervisor iriay reflect the fact that ^poor training 
placeS^e^ large burden on him. The readiness of his unit is reduced by a 
poo4clj?^rained person and he must take much of his time to bring this 

The higher return rate may also be due to the 
"^ract tha.irthe supervisor is more accustomed to administrative tasks and 
'^.that.he^J^^ more time than the trainee for "paperwork." Despite this 
t#^e^d|ffere'n in return rate between trainees and supervisors, strong 
■reasi^^/exist for including both ,the .trainee aijjd supervisor in post 
formal] training feedback. The trainee has a more up-to-date knowledge 
of the existing training in school and has direct knowledge of his own 
performance capabilities for job tasks. In theory, at least, he can 
provide more accurate data than the supervisor on certain training 
problems. Another reason for interrogating both groups is that super- 
visory personnel are often already burdened with a great deal -of paper- 
work, and sharing the feedback load between supervisors and trainees, 
;win relieve some of the supervisor's workload. The post formal training 
feedback task will tftus be less onerous for Fleet personnel. 

. ._Sinc5,.the '^expectation" factor improved the return rate for trainees 
the /pr^cti ce* of mailing advance letters to trainees should be followed 
i«. I^ny j nit lal effort for obtaining feedback information. However, a- 
bettej: procedure for. the continuing feedback effort would be to discuss 
JJtife, fefedback function with the student prior to his departure from 

school .^^ The materials could also be shown and explained to him at that 
J*ime^J)ut the materials should not be sent with him to th? duty station. 
^ fiiscussioi^s with training personnel indicate that. sending the materials 
f With. |he /trainee to his new duty station has often resulted in very low 
';;r4tuj^ rates. For .this reason, the evaluation' materials should be sent 
at I3^e time they are to be completed. Sinde the return rate from super- ' 
vt?ars was increased only slightly by advance notice, the "expectation" 
^Tefter- for supervisors can be omitted. * 

' ' .. .. . tS 

Joint administration (both 'the trainee and his supervisor) produced 
a higher return rate than single adminlstratlpn which was accompanied by 
higher accuracy and more comments. Although these features augur well 

^^'^h ' ■ • 'C ' 
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for a poljcy of joint administration, they are countered by information 
gleaned from follow-up letters to trainees responding in this joint 
administration mode. In several instances, there wefe clear indications 
that the trainee had actually filled out both his questionnaire and that 
of his supervisor. The fact that some trainees revealed this information 
could mean that numerous trainees were required to fill out the supervisor's 
form. For this reason, ^"either the supervisor or the trainee should be 
contacted, but both should not be contacted unless the number of graduates 
in a school is small and toth are needed to .provide reliable information. 
If the trainee fills out the supervisor's form in such single administration 
(or vice versa) , duplicate information wjll not be collected. 

Return rate differences were small for groups who had different 
time periods from graduation prior to the mailing of questionnaires; 
therefore, this measure i^rovides little basis for selecting one time* 
period over another. However, follow-up letters to persons who were, 
more than nine, months from graduation prior to interrogation often 
indicated that the respondent believed he should have received the 
materials earlier. Early iifterrogation also allows for earlier identi- s 
ficatton and correction of problems.^ Some minimum time, however, is 
required on the job to enable the trainee to evaluate his performance; 
to determine the adequacy of his training for this performance; and for 
the. supervisor to evaluate the trainee's performance. Probably three 
months would be ah optimum balance of these different factors. However, 
in the Navy, many graduates spend the first three months of their tours 
mess-cooking or op^other details unrelated to their training. In addition, 
leave often occurs after school and prior to assignment to -the duty 
station. All of these factors taken together indicate an optimum period . 
.of between five ands seven months following graduation prioir to interrogation. 

The novel and easy card-sort response required for the mail -out 
card-sort instrument was expected to produce a high rate of return; 
however, the data did not support this expectation. Although slightly^ 
higher than the return rate for the long questionnaire, the difference 
in return rate (1 percent}"was top small to argiie for adoption of this 
instrument and procedure. Even though the card-sort method successfully 
produced data about Trainee Performance Upon Arrival From School, which 
corresponded closely _to that of the long questionnaire {r=*89), it 
provided only partial data on Task Frequency and none on Task Critical ity. 
If additional scales such as Task Frequency or Task^ Critical ity had been 
iriclilded, sorting of cajwJfe would have become a cumbersome task and this ^ 
would'^negate the' efxpected advantage of the easy response. Ambiguous 
task statements tended to cause more problems with this instrument than 
with the l^ng questionnaire. -This suggests that the response o'f card- 
sorting sometimes occurs without enough time taken for analysis and' 
r?fle'ct1on. For these reasons, the card-sort technique is not recommended 
for general post formal tr'aining feedback purposes.- ,r \ / * 

• - , ' ' ^ . \ ' , 
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■ Having* both CHticality of Task and Performance of Task Upon 
Arrival From School scales on the short and long questionnaires provided 
an opportunity to calculate a derived Training Adequacy Index.. This 
involved subtracting the Criticjility ratings for an item from the Perform- 
ance ratings for the same item/ Items with high Performance ratings and 
low Critical ity ratings thuso>6ceived large values and items with low 
Performance ratings and high Criticality ratings received small values 
on this derived scale of Training Adequacy. A similar, procedure has 
been used.4)y Siegel, Schultz, and Federman (1951) to identify OYertrainirrg 
and undertraining in four Navy ratings. When this procedure was applied 
to data from the long questionnaire, it assisted in the identification 
of areas of known undertraining and, to a lesser extent, areas of known 
overtraining. However, in structured interviews, items which had the 
derived obaracteristic (*f "un^leJrtraimrfg"' oftdn did^rlot ^-Hcit'a strtilar*'" 
response from the interviewee when he was directly asked if more school 
training was needed for the task. An example of this was the item 
"Conduct emergency destruction of classified material" which persons 
interviewed face-to-face almost unanijrously believed was best learned on 
the job. Conversely, some areas where the derived score indicated 
oyertraining,were not judged as overtrained in school when the question 
was asked (e.g. , "Changing teletype tapes an<J ribbons"). Since the 
respondents in the interviews could readily express opinions on training 
adequacy for a task, and since these opinions were often largely indepen- 
dent of their Criticality and Trainee Performance ratings, it is important 
that both trainees and supervisors be asked to directly rate the adequacy 
of training rather than to rate the two factors of Criticality and 
Performance "of Task Upon Arrival From School. However, retention of the 
Frequency scale is recommended to aid in identifying obsolete equipment 
and tasks. The relatively easy rating of Task Frequency may 'also serve 
the function of getting the item solidly into consideration by the 
respondent prior to the more difficult rating of Training Adequacy. We 
expect, that the Frequency rating will not inaccurately bias the rating 
of Training Adequacy. 

Open-ended questions were used for two different purppses in the 
instruments administered. For both long anu short questionnaires, they 
were. included to elicit, new material that should be included in the 
school curriculum. Our recommended instrument and procedure also 
utilizes an open-ended question for this purpose since it is critical 
that a way be provided for identifying new training needs. For example, 
material that is^just now being integrated into the Radioman "A" School 
curriculum was -mentioped as missing from the curriculum by about three 
percent of the respondents. This implies that consideration must be 
given to ideas for new training if only a small percentage of persons 
report it. The. other pyrpose of open-ended questions was to get at the 
specific problems the short questionnaire could not identify with the 
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rating scales. As was mentioned, the success of this procedure was 
limited and we reconmend the use of a long form which mentions the 
specific situations, thus allowing all respondents to make judgments on 
possible training problems. 
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SECTION y . . ' 

. RECOMMENDED FEEDBACK INSTRUMENT AND PROCEDURES 

The long questionnaire used in the present study accurately identified 
^_the known problems of the old Radioman "A" School curriculum and provided'"' 
almost identical 'data to that obtained from more expensive face-to-face 
interviews. For these reasons., it is recorrmended that such mail-out 
• questionnaires, based on specific pob tasks, be adopted throughout the- 
Navy as the major method for obtaining post formal trainins feedback. 
This section describes the recommended instrument and procedures which 
are modified forms of the highly successful instrument and procedures 
used in, the present study and described earlier in this report. The 
modifications make' the instrument even more effective and simplify its 
administration and use. ' . / ^ 

The proposed feedback/curricuJNjm revision cycle requires 18 months 
plus the duration of the training course. In ^tddition, initial instrument ■ 
development may take from one to five weeks with the longer time required 
if job task .statements do no± already exist and the existing curriculum 
must be converted to descriptions of the job behaviors for which the 
school provides training. 

A minimum number of personnel would be required to develop these 
instruments and carry out these procedures. It would probably not 
require the full time of one person involved with curriculum devefopment 
except, during the initial instrument development stage and the data 
analysis phase which occurs after al] questionnaires have been re,tumed. 
Typing, printing, and clerical .services will be requi.red for brief *^ 
periods. Once .the -data are analyzed, the resuTtant inforination on • 
training problems would be fed intov.the'noiTnal curriculum revision 
process and the person who develops and admiaisters' the instrument wou.ld 
then be- free to devote full time to this revision prpcess. . . 

The recommended procedures are generally straightforward and require 
no' particular -skills or training of the person assigned tcT carry -th^m 
out. However, .if this person enjoys data tabulation and manipulation, 
so much the better. A -companion report (TAEG Feedback Manual, June 
1975) is written' for use"by relatively unsophisticated" traioing personnel" 
to enable thern to prepare and administer the instruments and" to analyze 
and use the ddta. * , ( 

FORMAT AND PREPARATION OF FEEDBACK' QUESTIONNAIRE* ' . 

« * 

The basic form which will hake up th'e bOlk of the questionnaire, is 
the next" to last page, -of each of the questionnaires presented in appendixes. 
J and K. It includes brief instructions, space foK eight. job tasks, and 
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two scales 6n which each of the items are to be rated by circling a 
number from one to five.. (IfN)ptical scoring is available, numbered, . 
dotted parallel Jines could be substituted for these numbers.) The 
"^Frequency of Task" scale is nearly identical. to thatojsed in the present 
study. No respondent reported any particular difficulty with this . 
scale. However, respondents often felt it difficult to rate a. task on 
the Critical ity and Performance scales if they had marked the task as 
"Never Performed" on the Frequency scale. In fact, despite instructions 
not to do so, they often skipped these offter scales rather than make 
what they believed to be inappropriate estimates about the critical ity 
of the task and the trainee's ability to perform the ta^.. On the new 
form; the respondent is given the option of skipping the Adequacy of 
School Training fof- This Task scale .if the task is rated as Never 
Performed." . , 

The Adequacy of School Training for This Task scale differs from 
the previous instrument which asked .for Criticality of Task and Perform- • 
ance of TaskUpon Arrival From School. As was mentioned in the previous 
section, it is expected that this will provide more useful data since 
structured interviews often indicated that 'supervisors and trainees made 
judgments about the adequacy of school training which were relatively 
independent of their ratings of Critical ity of Task and Performance of 
Task Upon Arrival From School. 

We recommend the basic job task form be used without change. 
Enough copies of this form should be reproduced to accofnmodate all of. 
the specific job tasks to be included in the questionnaire. These job 
task statements should be written in terms of observable behavior. If a 
job task analysis has been conducted, these statements will already ^ 
exist and preparation of the main body of the questionnaire can be -done 
very quickly. Before,.^4iia'myping, however, it is essential that several 
persons at the tr^fiifig unit, such as recent school graduates, ins.tructbrs 
and recent trart^fers from operational units, read these items to^help 
identify arty possible problems that might produce misinterpretations of 
. the task statements. . We suggest that these persons be asked to rate 
Task Frequency and Training Adequacy for each of these items in a structured 
interview similar to that which was conducted J n the present study. 

If job task statements are not available, the preparer of the 
instrument must convert areas of the existing curriculum into such task 
statements. This conversion of a curriculum to task statements will add 
much time to <iuestioTinaire development and wilt? require even more structured 
'interviews with personnel for the purpose 6f editing task statements 
prior to. typing the final form of the questionnaire. 

In most circumstances, supervisors will receive 125 .questionnaires 
similar to the one in appendix J, "find another 200 questionnaires 
(appendix K) will be mailed to trainees. Cover letters and biographical 
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data sheets for these questionnaires should be modifie^l from the forms 
in the appendixes to suit the needs and purposes o*f each training activity. 
One critical feature that should be retained in the cover letter is a 
deadline for return of the instrument. It is hoped that the instructions 
^nd open-ended questions can be reproduced directly from these appendixes 
jjflthout modification. • * 

' * ' \ 

^ If the number of^sk statements exceeds 200, twp- different question- 
naires should be made, vrith each containing half of the statements." 
This now vri 11, require tvrice as many persons to be sampled in order to 
obtain information about the course. - • ■ 

^ADMINISTRATION PROCEDURES • . 

If more than 325 trainees graduate in a six-month period, then 200 
of these should be selected randomly from the graduation rosters of the 
last six months for mailing of- trainee questionnaires and 125 should be' 
randomly selected for supervisor questionnaires. The randomization is 
important since it will minimize biases in the questionnaire data which 
might arise from improper sampling of different ability groups, or 
different dUty stations. The above numbers are predicated on the assump- 
tion that less than 200 task statements, are used on one questionnaire. 
If two questionnaires are used, twice as many persons- fff -available) - 
will be need.ed in each of the above groups. 

The questionnaires should .be mailed six months after graduation, 
with one month or more leeway. The supervisor forms should be mailed to 
the Commanding Officer at the trainy s duty station with instructions 
to forward the form to the "Supervisbr of Seaman (Name)." The trainee 
form can be mailed directly to the trainee at his duty station. Both 
letters should include a self-addressed envelope for'retam of the 
completed questionnaire. Two weeks prior to mailing the questionnaire 
to a trainee, the trainee should -te sent a letter that alerts him.to the 
inminentarrival of the questionnaire and its purpose, and that requests 
him to consider the adequacy of his training during the waiting'period. 

- If the trainee does not return fiis completed form within one month 
after mailing, a reminder letter (appendix L) should be mailed directly 
to him and not to his Conmanding Officer. No reminder should be sent to 
th6 supervisor who does not return the; questionnaire since he may have 
recently returned one for another graduate (see cover letter in appendtx 
J). Only the supervisor questionnaires will be mailed to the Commanding 
Officer and this will reduce handling of materials by personnel at, 
operational units. Every effort should be made to minimize the burden 
that post formal training feedback places on persons in supervisory 'and ^ 
conmand positions since thej,' often are already /flooded with paperwork. • 
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With an abundance of graduates, it wouTd be possible to mail more 
questionnaires than the initial 200 to trainee? and 125 to supervisors. 
This should be done if at becomes apparent that the return will fall 
considerably short of the goal of 75 trainee forms and 75 supervisor 
forms. If less than 325 trainees graduate in a six-month period, then 
at least some forms can be inailed to both tfte trainee and his supervisor, 
thereby increasing the .number of returned questionnaires." With less 
than 200 graduates, all trainees and all superv-isors of the trainees 
^ could be sampled. This sh'qiild result in more returned questionnaires 

'for 'supervisors than for trainees, but this imbalance is less important 
•^^than obtaining the 150 returned forms needed to provide reliable, repre- 
* sentative.data on the rating scales. If the school has a very small 
. number of graduates, the period of interrogation can be extended from 
SIX months to a year or longer in ord^r to build up a^satisfactory 
number of returned questionnaires. In .additjoij, more intensive follow- 
up procedures (e.^., phone calls, a second questionnaire, contact of 
supervisors) can be used to obtain returned questionnaires. 

The sample of graduates should be selected from graduating classes 
over a six-month period. One reason is that temporary school problems 
, will not have as much influence in the results. ^Iso, over a short 
period, of time an unrepresentative samtrHrrg-of duty stations could 
easily occur. Another advantage of a sfx-month interrogation period is 
that It greatly reduces the workload involved with mailing. question- 
naires and recording the returned data. Only about eight expectation 
letters, 13 questionnaij^ and five follow-up letters would need to be 
mailed each week and only^data from about six returned questionnaires 
recorded. This would require a small amount of time and allow opportun- 
ity fdr checking the out^gping materials- and- insuring that they are free 
of defects. • 

PROCEdURES FOR ANALYSIS AND UTILIZATION OF QUESTIONNAIRE DATA • 

A notebook should be prepared with a page for each itep^ on the 
questionnaire. This page could take the form of;the example in appendix 
M, which has space for maintaining separate records for Trainee Frequency 
and .Training Adequacy and Supervisor Frequency and Training Adequacy. 
When a questionnaire is>eturned, an identification number should be 
assigned to it. This identification number can be written on each item 
page in the columns that- qorrespond to that person's ratings on Frequency 
and on Adequacy. Any comments which are written regarding training for 
an Item should be written on the back of the particular notebook page 
for that Item and the identification number included to identify the 
origin of the conment. When a returned questionnaire has a response to 
the open-ended question asking for new material that should be included 
in the curriculum, a new page for the notebook should be made on which ' 
this response is written along with the identification number of its 
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aut4ior. A separate page should l?e added for each new item mentioned. 
As future questionnaires arrive that ^Iso mention this item, their • 
identification numbers should be added to this page* 

After all questionnaires have been mailed and follow-up procedures 
fail to produce more return's, anfl if |he; number returned are at least .50 ^ 
for the trainees and 50 for the supervisors (if less, mail more question- 
.naires), the data analysis may begin. Separate statistical means should 
be calculated for each item for trainee ratings of Frequency, tjainee 
ratings of Training Adequacy, supervisor ratings of FrequeJicy, and 
supervisor Ratings of Training Adequacy. The numbers of Training Adequacy 
ratings will differ softewhat across p&ges (items) beca.use of the option 
respondents have of skipping that scale. It is important to calculate 
;the mean with the actual nunier of ratings given for* the item and'nort 
with the total number of trainees or supervisors who returned question- 
naires. ^ ' 

. In addition to these four means, an additional mean should be 
calculatedyfor each item sheet which is the average of the average 
rating of Training Adequacy for all trainees and of the auerage rating 
*of Training Adequacy for all' supervisors. After this "overall rating of 
Training Adequacy" has been calculated for all items, the pages for the 
itons should be reordered!' with high values of this average at one ^nd 
andjow values at the other. . . ' . 

The 10 percent of items at each end of this reordered stack are , 
prime candidates for an investigation of training and possible curriculum 
revisions. Unless other factors argue very strongly against it, those 
'itens which are closest to the "task requires much more emphasis in 
school" end of the Adequacy of Schoolf Training scale should be allotted 
more training emphasis. Similarly, the items which'are closest to the 
"greatly reduce or eliminate training for this task" end of the scale 
should have the time and other resources given to their training reduced. 
•When reduced training emphasis is recommended and ratings of Tssk Frequency 
indicate a very low rate of performance, the situation is probatJly one 
of irrelevant training. More drastic curriculum.changes may be called 
for than in the overtraining condition where reduced training is recommended 
but frequency of performance of the task is moderate or high. 

Many factors operate to bring average ratings of Training Adequacy 
toward the midpoint of the scale and average ratings of highest and 
lowest items 'may differ by only a little more than one point. This 
should not be used a's an, excuse to refrain from curriculum revision. 
Th^ standard error of an average of 150 item ratings will be less than 
.1. If scales w ere markfi^andomly, the chances of a difference of one 
scale un|t or nwre betwe^^ighest and lowest items would b^ less than 
one in arte thousand. Thus, the differences in average Training Adequacy 
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between 'the top and bottom 10 percent of items can be assumed. to be real 
differences and not chance occurrences. Unless t'he school is an exceptional 
one, most itehis falling within the' 10 percent margins can and should be 
acted upon.' If personnel and time are not available to correct training 
for all of the top and bottom 10 percent of these items,. thos6 with the' 
most extreme . average ratings of Training Adequacy should be attended to , ; 
first. ' ' ' 

A factor that might; countermand the recommendations of these average 
.Training Adequacy ratings would be if a very lefrge percentage (perhaps 
■ 70 percent) of the total sample skipped the Training Adequacy scale for 
the item. Under such circumstances where a task is performed by very 
fsw graduates, a recommendation to greatly increase training emphasis 
would be suspect. Other j'actdi-s t^iat could counter Training Adequacy ^ 
reconmendations would be feedback data from other sources. If. equipment 
breakdowns are numerous for a piece of equipment, an average Training 
Adequacy recommendation to reduce maintenance training for theMtem » 
would require much consideration before implementation. 'Generally, 
however, the Fleet personnel are as much aware of these other factors as 
training personnel, and in most instances, these average ratings of 
Training Adequacy can be heeded. In any curriculum modification, the " 
comments and recommendations included with item ratings should be given 
much consideration. . 

Job tasks recommended for addition to the curriculum' may be included 
if enough persons (perhaps 10 percent of respondents) recommend them and 
other sources agree to this need. A small perpentage such as this' can 
be actecf upon since many other persons would agree to the need but did . ■ 
not think of it at the time of questionnaire' completion. -Other, less . 
verifiable, recommendations can be included in the revised questionnaire \ 
to be used in the next round of evaluations to determine whether or not* . ' 
they should indeed be added to the curr.iculum. . . ; 

• IdeaTlj, the^rexlsi on^ should be made and- the new curriculum ' 
Implemente.d witi^Tn six months following the analysis of questionnaire 
djta^~ Whatever the period required for 'curriculum revision, the' field 
evaluation process can. be repeated beginning six months after the first 
trainees graduate from the new curriculum. Durin'g this six months, 
between graduation and questionnaire mailing, the questionnaire can ^e " 
" brought up-to-date to include new material added to the curric^ilum and 
material under consideration for future use. As classes -graduate, the 
randomization 'process can begin to select persons who are to receive 
questionnaires six months later. If possible, persons selected to , - 
receive trainee forms should be shown the questionnaire and made aware 
that they will be expected to respond to the inquiry. 
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After six months- of qyesttpnnaire mailings and when it is determined 
. that no more questionnaires^i^are .forthcoming, the data analysis and 
curriculum revision process can begin agaiii. This cycle of six-months, 
ot mail-out interrogation— six months of dl^a analysis and curriculum 
revi$ion~durati.on of revised course-six moflths of feedback questionnaire 
revision and respondent selection-rback to six-months of mail -out interro- 
c«r!!l!/i^^ ^^^^ Pl"s the duratidn pf the training course. 

• Fpr.most Navy course!^, the tiige period wduld be slightly'less than two 
years between mail-out interrogations. .With, today's rapidly changing . 
SyTr'h!^LSrj°'' alniost guarantees that a new set of training problems 
will be ready. for identification. , • 

f^nm iLc^^^*-?" the indications of need' for training change resulting 
from these mail-out i;istruments and procedures, internal sources of 
training problems will also be providing input to the curriculum revision " 
from hn;*, ^'""umably much agreement will exist between the recomnendations 
S r^SJ54°!^^'f • .^"ly.tf^e d^t^ ^0"! the. Fleet, however, can establish 
for certairr 'that a training objective is being adequately met, and, 
perhaps of .morejmportance,. only this data can .indicate whether existing 
training! objectives appropriate ones. ' 

« 

SUMMARY Of RECOMMENDED PROCEDURES • ■ 

fnr nht.4nl^T S^^^J^^"!'^'^^ ' °" specific job tasks, i recoiTOended 

for obtaining feedback data in all Navy schools.. f - 

. ■ „ "Rat-ings should be -obtained for each specifn'c job task oh 4e 
frequency of theO:ask and the adequacy of school training for the task. 

.. : An open-ended question should be included- to get at missing ^ 
training object>ves. v , ' • . 

. Questionnaires shoi(M^ mailed to. both tr<iinees and supervisors, 

. The sample to be teskiT^U be selected randomly from all 
graduates over a six-month p^rTod. \ \ . 

Questionnaires should be ma\led six months after graduation. ' 

■ • % Enough questibhnaires should be mailed to obtain 75 returned 
from trainees/^d 75 returned from supervisors: 

r' ^- Questionnaires fof' supervisors' should be addressed 'T;o their' 
Commanding Officers. J > v 



1/ 

V 

trainee 



Questionnaires for trarinees should be mailed .directly to *he 
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Trainees should be notified that they will receive feedback . 
questionnaires. .This can be done by mail or, more economically, whil-e ; 
they are still in school. . . 

. Follow-up procedures should, be initiated one month ?fter ' 
mailing o"f.the questionnaires tn order to*jncrease the returpi rate.; 

Average ratings" of Frequency and Adequacy^of ScRool Training ^ 
for tasks should be calculated for ea2ft* questionnaire item. 

At least the t5p 10 percent and bottom .10 percent of the taslcs 
with extreme average Adequacy ratings should be reviewed and a revision 
of .the curriculum made where necessary. - 
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FIFTEEN GENERAL TASKS 'OF SHORT QUESTIONNAIRE . - 

1 . TYPE MESSAGES ON A TELETYPEWRITER KEYBOARD USING THE TOUCH TYPE 
METHOD. 

2. PREPARE TELETYPE TAPES OF MESSAGES WITH ROUTING INDICATORS FOR 
TRANSMISSION m AUTODIW FORMAT OR MODIFIED ACP.126 FORMAT. 

3. PROCESS OUTGOING MESSAGES. 

4. P'ROCESS INCOMING MESSAGES. 

5. MAINTAIN 'COMMUNICATION MESSAGE FILES. 

6. OBSERVE SAFETY PRECAUTIONs'^WHEN WORKING WITH ELECTRONIC EQUIPMENT. 

7. SELECT, SET UP AND PATCH TELETYPE AND VOICE EQUIPMENT. 

8. RESTORE MALFUNCTIONING SUB-SYSTEMS TO NORMAL OPERATION. 

9. PERFORM PLANNED MAINTENANCE SUB-SYSTEMS ACTIONS. 

10. OPERATE A FLEET MULTICHANNEL BROADCAST. - 

11. OPERATE SHIP-TO-SHIP AND SHIP-TO-SHORE TELETYPE CIRCUITS. 

12. OPERATE THE AN/FYA-71 (V) DSTE AUTODIN TERMINAL. 

13. OPERATE THE AN/FGC-73 (V) MULTIPLE ADDRESS PROCESSING UNIT. 

14. EFFECT DISTRESS COMMUNICATIONS USING THE RADIOTELEGRAPH MODE OF 
OPERATION. 

15. MAINTAIN SECURITY OF CLASSIFIED MATERIAL AND COMMUNICATION. 
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134 SPECIFIC TASKS OF LONG QUESTIONNAIRE 



1. TYPE PRELIMINARY CALLS AND OTHER TRANSMISSIONS ON A TELETYPE KEYBOARD, 
USING THE TOUCH TYPE METHOD. 

2. CUT TEl^OPE TAPES OF MESSAGES DESTINED FOR TRANSMISSION IN EITHER 
AUTODIN OR MODIFIED -ACP-126 FORMAT. ' 

3. ■ CHECK THE ACCURACY Ol^PREPARED TAPE. . - * : 

4. READ'PRECUT TELETYPE 'tAPES CONTAINING NO PRINTING 0J{ THE JAPE. 

5. CORRECT PRECUT TELETYPE TAPES. • „ 

6. ' CHECK THE TAPE HEADING BETWEEN. FORMAT LINE FOUR AND EOM. - 

7. PREPARE TAPES FROM ORIGINATOR'S ROUGH DRAFFSx 

8. PREPARE HEADER REQUIREMENTS. ^ - ■ . > 

0 

9. IDENTIFY COMMAND, COLLECTIVE, CONJUNCTIVE AND GEOGRAPHIC ADDRESS 
GROUPS. 

10. COMPLY WITH UNIT'S OPERATIONAL CHAIN OF COMMAND WHEN PROCESSING 
^MESSAGES. . • ' ^ A 

11. COMPLY WITH UNIT'S ADMINISTRATIVE CHAIN OF COMMAND WHEN PROCESSING 
' MESSAGES. 

12. CHECK THE VALIDITY OF THE RELEASING OFFICER'S SIGNATURE ON EACH 
MESSAGE. 

13. ' HANDLE EACH MESSAGE IN ACCORDANCE WITH THE PRECEDENCE ASSIGNED BY 

THE DRAFTER. * 

14. ASSIGN TO EACH MESSAGE A DATE-TIME GROUP. . 

15. LOG OUTGOING MESSAGES IN THE CENTRAL MESSAGE LOG. 

■ . ' • ■ J - 

16. DISTINGUISH BETWEEN VARIOUS TELECOhWUNICATION METHODS OF MESSAGE 
DELIVERY. • ' . . \ 

17. DETERMINE THE METHOD OF MESSAGE DELIVERY to. BE EMPLOYED AND THE - 
FORMAT REQUIRED. 

72. ' ^ ' 
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18. ENCODE/DECODE ADDRESS GROUPS. 

19. ASSIGN CALL SIGNS, ADDRESS BROUPS, & ROUTING INDICATORS (AS REQUIRED). 

20. identify' incorrect naval activity short TITIeS '& CHANGE THEM TO CONFORM 
TO THE PLAD. - < ■ 

21. IDENTIFY ELEMENTS OR^AOTpDiN HEADERS. 
.22. PLACE MESSAGES IN AUTODIN FORMAT. 

0 

23. CORRECT FORMAT ERRORS IN MESSAGES FORMATTED lAW JANAP-128. 

24. CONVERT MESSAGES IN AUTODIN FORMAT TO MODIFIED -ACP-1 26 FORMAT. 

25. PLACE REUY INSTRUCTIONS ON MESSAGE HEADINGS. 

26. CHECK EACH MESSAGE FOR PROPER aASSIFICATIO^ CRITERIA INCLUDING. . ■ 
SECURITY WARNINGS IN FORMAT LINES 2 & 4. 

27. VERIFY GEOGRAPHICAL LOCATIONS "WREN PROCESSING OUJGOING MESSAGES. 

28. idefftify category of precedence.' 

29. comply with handling time objective for each precedence category." 

30. verify existence of classification and d0wngrapin6/declassificati0n 
markings; 

31. verify existence of standard subject identification code (ssic). 

32. screen- message headings for m£"5sages addressed to addressees on 
guardlist. 

33. record time of receipt of each message addressed to guardlist. 

34. alert personnel when flash message is received & perform the 
prescribed processing. actions for flash" traffic. . 

35. DETERMINE THE INCIDENCE OF RECEIPT Of\jPLICATES OF MESSAGES 
PREVIOUSLY RECEIVED. 



IF WLICATEJ 



36. RECORD, THE ORIGINATOR^ AND DATE-TIME GROUP 'OF tACH. MESSAGE RECEIVED. 

37. USING AN INTERNAL ROUTING GUIDE, INDICATE THE INTERNAL ROUTING 
. NECESSARY FOR EACH MESSA^ ADDRESSED TO GUARDLIST. 
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38. MAKE REPRODUCTION OF EACH MESSAGE ADDRESSED TO GUARDLIST IN SUFFICIENT 
.QUANTITIES TO. SATISFY THE INTERNAL. ROUTING INDICATED. 



39. MAKE INTERNAL DISTRIBUTION OF MESSAGES RECEIVED IN ACCORDANCE WITH 
■ THE INTERNAL ROUTING INDICATED. 

40. IDENTIFY CALL SIGNS. 

41. ' IDENTIFY BROADCAST MESSAGES. 

42. MONITOR PAGE PRINTERS THAT ARE ELECTRICALLY CONNECTED INTO THE FLEET 
RADIOTELETYPE BROADCAST SUB-SYSTEM. ^ - * 

43. CHECK-OFF BROADCAST NUMBERS BY INDICATING THE CLASSIFICATION OF EACH 
MESSAGE PASSED ON THE BROADCAST. 

44. DETERMINE THE INCIDENCE OF MISSING BROADCAST NUMBERS, BY CONTINUOUS 
NUMBER CONTINDITY CHECK. 

45. RECEIVE MESSAGES VIA THE* FLEET BROADCAST. ■ " 

46. MAINTAIN BROADCAST FILES. 

47: FILE ONE COPY OF ALL FIRST RUN TRAFFIC IN THE BROADCAST F-ILE IN 
BROADCAST NUMBER ORDER. 

48. DELIVER ONE COPY OF ALL FIRST RUN TRAFFIC TO THE BROADCAST TRAFFIC 
CHECKER. 

49. IDENTIFY SPECIAL CATEGORY MESSAGES. ' . ^ . . . 

50. IDENTIFY GENERAL' MESSAGES & THEIR SERIAL NUMBERS. 

51. IDENTIFY READDRESSED MESSAGES, , * ' 

* * 

52. FILE TOP SECRET AND SPECIAL CATEGORY MESSAGES IN A CRYPTQCENTE^ FILE 

IN DATE-TIME GROUP ORDER, AND CONSTRUCT A FILLER, FOR EACH MESSAGE .FILED. • . 

53. FILE GENERAL MESSAGE IN A GENERAL MESSAGE FILE, SEGREGATED BY GENERAL 

• MESSAGE TITLES IN SERIAL NUMBER ORDER: CONSTRUCT A FILLER Rfifi^CH " 
MESSAGE. SO FILED. 

: ■ s i 

54. PREPARE A FILLER FOR EACH READDRESSAL MESSAGE THAT IS PROCESSED. 

55. FILE ALL MESSAGES (OTHER THAN TOP SECRET, SPECIAL CATEGORY, AND GENERAL 
MESSAGES) AND FILLERS IN THE COMHUN I CATI0NS CENTER FllE IN DATE-TIME 



GROUP ORDER. 
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56. CONDUCT ROUTINE DESTRUCTION OF CUSSIFIED. MATERIAL. — 

57. . CONDUCT EMERGENCY DESTRUCTION OF CLASSIFIED MATERIAL. , 
■ 58. FILE MONITOR.ROLIS, RADIO LOGS, & SEND/RECEIVE LOGS. 

59. DETERMINE THE MEANING OF PROSIGNS. •' 

60. DISTINGUISH BETWEEN PRO^lfiNS & OPERATING SIGNALS. 

61. ENCODE/DECODE VOICE CALL SIGNS.' 

62. DISTINGUISH BETWEEN SHJP/SHIP & SHIP/SHORE TELETYPE CIRCUITS. 

63. DISTINGUISH BETWEEN THE THREE MAJOR COMPONENTS OF A NAVCOMMSTA IN 
REUTION TO THEIR CONTACT WITH THE SHIfBOARD SHIP/SHORE OPERATOR. 

64. ENCODE/DECODE OPERATING SIGNALS. 

65. INITIATE AND ANSWER PRELIMINARY CAtLS. • 

66. TRANSMIT MESSAGES IN AUTODIN ANO MODIFIED ACP-126 FORMAT IN. THE 
ORDER OF THE PRECEDENCE ^SIGNED. 

- 67. PLACE TAPES IN BACKLOG BIN BY." PRECEDENCE. ^ 

68. .REQUEST, AND REPLY TO REQUESTS FOR REPETITIONS AND CORRECTIONS. 

69; PROVIDE RECEIPT FOR TRANSMISSIONS AXD MESSAGES, AFTER ENSURING THAT 
■THEY ARE ERROR TREE.- . 

"t 

70. LOG ALL'TRAI^SMITTED'AND RECEIVED MESSAGfS IN THE SEND AND RECEIVE 
•LOGS,' RESPECTIVELY. , . ' ' 

71. "AFFIX A'TRANSMtSSION OR RECEIVE ENDORSEMENT tO EACH MESSAGE 
TRANSMinED AND^ReCEJVEP, RESPECTIVELY. - . ' 

72. PERFORM PAPER, TAPE AND RIBBON CHANGES, AS NECESSARY. ' 

73. LOG HME ENTRIES IN RADIOTELEGRAPH. LOg'eACH TRANSMISSION. 

74. PERf^tJRM JHE RESCUE OF A PERSON IN CONTACT WITH A LIVE CIRCUIT. 

75. PE^ORM THE FOLLOWING FIRST AID PROCEDURES: MOUTH-TO-MOUTH 
RESUSCITATION, BACK-PRESSURE ARM-LIFT AND BACK-PRESSURE HIP-LIFT 
ARTIFICIAL RESPIRATION, JREAIMENfFdR SHOCK, AND TREATMENT. f^OR BURNS 
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76. OBSERVE SAFETY PRECAUTIOfiS WHEN WORKING WITH ELECTRONIC EQUIPMENT. 

77. PERFORM THE NECESSARY SAFETY PROCEDURES FOR GOINff' ALOFT. 

78. IDENTIFY NAVAL COMMUNICATION EQUIPMENT BY MEANS OF THE .ioiNT 
ELECTRONICS TYPE DESIGNATOR SYSTEM. ' . 

79. ADJUST THE OPERATING CONTROLS OF THE AN/WRC-1 TRANSCEIVER. 

80. ADJUST THE OPERATING CONTROLS OF THE AN/URC-9 TRANSCEIVER. 

81. ADJUST THE OPERATING CONTROLS OF THE AN/UpC-1 TELETYPE TERMINAL 
• EQUIPMEJ^T. • ■ ' 

^ 

82. ADJUST THE OPERATING CONTROLS OF THE AN/SGC-IA TELETYPE TERf^INAL 
EQUIPMENT. , ^ ... 

83. ADJUST THE OPERATING .CONTROLS OF THE AN/SR6-20. 

84. ADJUST THE OPERATING CONTROLS OF THE R-1051 RECEIVER. 

k 

85. ADJUST THE I^RONT PANEL CONTROLS OF .THE AN/URT-23 TRANSMIHER. 

86. ADJUST THE OPERATING CONTROLS OF THE AN/URA-17 TELETYPE CONVERTER. 

87. ADJUST THE OPERATING CONTROLS OF THE AN/WRT-2 TRANSMITTER. 

88. ACTIVATE UNCOVERED MF/HF VOICE SUB-SYSTEM (TYPE Y). 

89. ACTIVATE UNCOVERED UHF VOICE SUB-SYSJEM (TYPE U). 

90. ACTIVATE- COVERED UHF/VHF VOICE SUB.-SYST£M (TYPE R). 

91.1 ACTIVATE UHF ORESTES COVERED SIMPLEX SUB-SYSTEM (TYPE B). 

92. \ ACTIVATE UHF OREST-ES COVERED ^UPLfX SUB-SYSTEM (TYPE C), 

93. ACTIcVATE MF/HF .ORESTES COVERED SIMPLEX SUB-SYSTEM (TYPE p). v 

94. ACTIVATE MF/HF ORESTES COVERED DUPLEX SUB-.SYSTEM (TYP^ G). 

95. ACTIVATE ORESTES COVERED SHIP-SHORE MULTIPLEX TERMINATION (TYPE P) 
P6. ACTIVATE COVERED FLEET MULTICHANNEL BROADCAST, RECEI.VER (TYPE N).. 
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97. DETERMINE THE EFFECTS OF THE IONOSPHERE ON SKIP DISTANCES AND SKIP 
ZONE, AND USE THE RESULTS OF TH-IS DETERMINATION AS AN AID IN 
ESTABLISHING A LONG DISTANCE COMMUNlP^j-ION- PATH. 

98. .DETERMINE THE EfI^ECTS OF THE I.ONOSPHERE ON SKYWAVE PROPAGATION AND 
MSE THE RESULTS OF THIS DETERMINATION AS AN AID IN ESTABLISHING A 
LONG DISTANCE COMMUNICATION PATH. ' . 

r ■ • . ■ 

99. TUNE ANTENNAS TO ACHIEVE OPTIMUM- TRANSMISSION. 

100. ' OPERATE THE KWX-8 CONTROL UNIT ON CRYPTO EQUIPMENT. 

101. CHECK SUB-SYSTEM OPERATION FOR INDICATIONS ^F SUBrNORMAL PERFORMANCE 

102. ISOLATE THE INCIDENCE OF SUB-NORMAL PERFORIWNCE TO IMPROPERLY 
ADJUSTED EQUIPMENT, MALFUNCTIONING EQUIPMENT, FAULTY PATCHES, OR 
FAULTY .COMMUNICATION PATH. 

103. RESTORE SYSTEMS TO NORMAL 'OPERATION BY CORRECTING' THE PATCH, 

' COMMUNICATION PATH SUBSTITUTION, OR BY READJUSTMENT OR REPLACEMENT 
' OF IMPROPERLY ADJUSTED OR FAULTY EQUIPMENT. ( 

104. LOCATE PM5 ASSIGNMENT ON T?E 3-M WEEKLY SCHEDULE. ' 

105. LOCATE THE MAINTENANCE REQUIREMENT CARD (MRC) WHICH PERTAINS TO 
THE.PMS ACTION ASSIGNED. 

lOe. LOCATE THE SPECIFIC EQUIPMENT ON WHICH THE PMS^ ACTION IS TO BE 
PERFORMED. 

1.07. PERFORM THE PMS ACTION CALLED FOR BY THE MRC. 

1.08. PAIfjt ANTENNAS. . ^ ' . 
109. ■ CLEAN ANTENNAS. ' ' . ^ ' ' 
no. PERFORM RESISTANCE CHECKS ON ANTE^INAS. 

111. PERFORM VISUAL INSPECTION OF ANTENNAS. 

112. OPERATE PAPER TAPE READER, CARD READER & SEND PORTION OF THE ; 
COMMON 'control 'UNIT OF THE AN/FYA-71(V) D3TE AUTODIN TERMINAL. 

113. CONDUCT CONTINUITY CHECK OF THE MULTIPLE ADDRESS PROCESSING UNIT 
SYSTEM EVERY HOUR ON THE HALF HOUR., 
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114. RESPOND "IN W 'PRESCRIBED" M/^NER 'TO ANY AlAfW CONDITION IN THE * 
MULTIPLE ADDRESS PROdfSSING UNIT SYSJEM. ^ . ^' ^ 

115. OPERATE JHE TT-Sag ^EP^ERFORATeB^l! ' ' . ^' 
116.. OPERATE THE^TT-331 (TWO ROWS OF THREE REPERFORATORST) . 
\17. MAIMT^'IN BROADCAST STATUS LOG. ^ ^' 

118. INITIATE/RESPONQ TO PRELIMI|iARY C/\LLS AS -THE NET CONTROL STATION. 

119. ' TRANSf^IT-^ARD TRAFFIC* ' • * 

120. aOSE .OUT SEND CHANNELS.. ^ 

121. . DELIVER ALL MESSAGE-TAPES RECEIVED TO THE MULTIPLE ADDRESS = 

PROCESSING UNIT SYSTEM OPERATOR.- .' , - 

•122. USE- THE INTERNATIONAL DISTRESS AND CALLING FREQUENCY 'AND OBSERVE 
THf SILENT PERIODS., 

123. RES'POND TO' DISTRESS, URGENCY, AND SAFETY SIGNALS, AND TO THE .FIVE 
ENEMY CONTACT ALARM SIGNALS EMPLOYED BY MERCHANT VESSELS JN TIME 
OF WAR. ' V - • 

124*. PROVIDE RECEIPT FOR, OR RELAY, IN INTERNATIONAL FORM, INTERCEPTED 
DISTRESS MESSAGES •AND.'ENEMY.TONTACT REPQRTS, USED BY MERCHANf 
, VESSELS IN TIME .OF WAft. 

125. PREPARE .A MESSAGE FOR TRANSMISSION BY RADIOTELEGRAPH 'iN PLAINDRESS, 
ABBREVIATED PLAINDRESS, AND CODRESS FORM. ?• 

126. OPERATE A RADIOTELEGRAPH CIRCUIT USING MILITARY PROCEDyRES AND EMPLOY- 
ING CALL' SIGN ENCRYPTION AND AUTHENTICATION. - 

127. USE THE COUNTERMEASURES AVAILABLE TO REDUCE THE EFFECTS OF JAMMING AND 
HARMFUL INTERFERENCE. , ' . / 

128. RECORD THE REQUIRED INFORMATION NECESSARY FOR REPORTS ^HEN 4:0WDITI0NS 
. ' of jamming or harmful interference ARE DETECTED.- . 

129. ENCODE/DECODE INTERNATIONAL JCA^ SIGNS. 

130. ENCODE/DECote TASK organization' CALL. SIGNS. • . ' 
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133. . GUARD" AGMNST ANY INCIDENCE^OF COMPROMISE OF CLASSIFIED MATERIAL 
OR ANY SECURITf VIOLATION. . . ■ . ' . 



V. 132. EMPLO^^PHYSICAL SECURITY MEASURES BY ADHERENCE TO THE ACCOUNTING, 
■ DISSEMINATION, & STOWAGE PROCEDURES PRESCRIBED FOR CLASSIFIED . . 
MATERIAL. • - • • . - ^ ' , . 

. 133. EMPLOY TRANSMISSION SECURITY MEASURES BY ADHERENCETO PRESCRIBED " 
^ TRANSMISSION PROCEDURES AND BY ALERXNESS TO AND REPORTING* OR 
DEVIATIONS FROM THESE PROCEDURES.. / 

. 134. PROJECT CRYPTOSECURITY BY THE-TROPER USE OF CRYPTOGRAPHIC MATEklAL, 
CRYPTO SYSTEMS, AND DELATED CRYPTO' MATERIAL.' 
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APPENDIX C 



BASIC RATING SCALE FORM USED ON LONG 
AND SHORT QUESTIONNAIRES 
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• APPENDIX L 



RECOMMENDED. REMIND^ LEHER 




■lie . 



From: Curriculum Update* Branch, Sdwol 

to: Recent " ^ School Graduate 



About one'month ago you were mailed materials for. evaluating the 

training at Schoo^f. It 1s most important that we receive 

your completed questionnaire in order that we my use the data in our 

continuing program to mak^ training of both* relev^int and 

effective; ^ " . ' • ^ 

If you did not receive the materials, please .contact us and we 
will raail another set. If you did receive* thpm, please complete' and 
reiufn them as soon as* possible. . ' * „ 

Sincerely, ' 

Chief Petty Offi^ber 
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, ITEM # 52 NUse principles of sky wave propagation to est^iblish a long distance 
V ' ciMimmication path. • - - ^ 



' TRAINEE 

FREQUENCY OF TASK' 



TRAINEE 



• A^OUACY OF SCHOOL TRAINING 
FOR THIS TASK 



SUPERVISOR 

FREQUENCY OFTASK* 



SUI^RVISOR 

ADEQUAC1? OF SCHOOL TRAINING 
^FOR THIS TASK 



L Never perfonrnfd* 

2. Seldom perfonned or only In emergencies. 

3. Performed raoflUity. 

4. Perfo^nved weekly. 

5. P»frfoRned dally. • 



•ADEQUACY ic^t be skipped If task is 
never performed. " > * 



' 2 



^ Task requires much more emphasis In school 

2. Training less thai adeqpate for task. 
Increase etnphasls. 

3. Training adeijuate for task. 

4. Training more \tm adequate for task, 
reduce emphasis.' 

5. Greatly reduce or ellnlnxte training for 
this task. 



4 ' 



L Ne>v performed.* 

2. Scldemperforoiedoronlyjnjunergenctes. 

3, Performed monlNy. , • 

4, Perfonned weekly. - • 

5. Performed dally. , * 



'ADEqUACY scale may be skipped If task Is 
' never ptrfonned. 



L Task* requires much more emphasis In schx) 
2» Training less than adequate for task, 
, Increase emphasis. 

3. Training adeqi/ate for task. 

4. Training more than ^et^^ate for task, 
reduce e«^hasl$. • 

5. Greatly reduce or eliminate training for 
^_ this task. 

i 
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